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The District of Columbia Public Schools (DCPS) has prioritized efforts to support students’ social and emotional 
learning (SEL) competencies, such as perseverance and social awareness. To measure students’ SEL 
competencies and the school experiences that promote SEL competencies (school climate), DCPS began 
administering annual surveys to students, teachers, and parents in 2017/18. DCPS partnered with the Mid- 
Atlantic Regional Educational Laboratory to study how the district could use these surveys to improve students’ 
outcomes. The study found the following: 

e Students’ SEL competencies and school experiences are the most favorable in elementary school and the 
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experiences when students need the most support. 

The trajectories of students’ SEL competencies and school experiences differed in different schools, to a 
similar degree as trajectories in academic measures like test scores. To understand why changes in SEL 
competencies and school experiences differ across schools, DCPS could explore differences in practices 
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schools with lower levels of positive change. 

Of the SEL competencies and school experiences in DCPS’s survey, self-management—how well students 
control their emotions, thoughts, and behavior—is most related to students’ later academic outcomes. 
Programs or interventions that target self-management might have the most potential for improving 
students’ outcomes compared to those that target other SEL competencies or school experiences. 

In statistical models designed to predict students’ future academic outcomes, SEL competency and school 
experience data add little accuracy beyond prior academic outcomes (such as achievement test scores and 
attendance) and demographic characteristics. Prior academic outcomes and demographic characteristics 
predict later outcomes with a high degree of accuracy, and they may implicitly incorporate the SEL 
competencies and school experiences. These findings suggest that DCPS would not need to use SEL 
competencies and school experiences to identify whether or not students are at risk of poor academic 
outcomes. 

Student, teacher, and parent reports on SEL competencies and school experiences are positively related 
across schools, but they also exhibit systematic differences, suggesting that some respondent groups may 
not be aligned in their view of SEL competencies and school experiences. These differences may serve as 
a tool to help DCPS target efforts to improve communication among students, teachers, and parents. 


Why this study? 

Students’ social and emotional learning (SEL) competencies—such as how well students persevere, manage their 
thoughts and emotions, and understand what others think and feel—have been shown to be related to many 
life outcomes and, importantly, can be shaped through education. Such SEL competencies rival cognitive 
measures (such as IQ) in predicting long-term outcomes, including educational attainment, health, earnings, and 
employment (Heckman & Kautz, 2012). They are also related to shorter-term outcomes, such as academic 
achievement (O’Conner et al., 2017). In addition, interventions can improve SEL competencies throughout 


grades K-12, suggesting that SEL competencies are a promising avenue through which education can improve 
the long-term success of students (Elango et al., 2016; Kautz et al., 2014). Evidence suggests that one way 
schools may be able to boost SEL competencies is by fostering a positive school climate—the tangible and 
intangible attributes of a school that support students’ development—including relationships among students 
and staff, school discipline, student engagement, and safety (SRI International, 2018). 


Inspired by this type of evidence, the District of Columbia Public Schools (DCPS) has prioritized supporting positive 
SEL outcomes (for example, perseverance) and has begun administering surveys to measure students’ SEL 
competencies and school climate. In its 2017-2022 Strategic Plan—a proposal that DCPS uses to outline key goals 
and hold itself accountable to the public—DCPS highlighted SEL competencies and school climate as key 
components in its priority to “educate the whole child” (DCPS, 2017). All six goals reflect this priority, either 
explicitly or implicitly. For example, DCPS has set an explicit goal that, by 2022, 100 percent of students will feel 
“loved, challenged, and prepared,” as measured by an index based on SEL competencies and school climate (DCPS, 
2017; see figure A1 in appendix A). Other goals—such as improving college and career readiness and increasing 
re-enrollment rates—are implicitly related to SEL competencies and school climate in that they might be furthered 
by better identifying at-risk students and boosting their SEL competencies and school experiences. To track 
progress toward its goals, DCPS launched annual surveys (developed by Panorama Education) in spring 2018 that 
collect information from students, teachers, and parents on SEL competencies and experiences that reflect school 
climate. 


Despite a strong research base supporting the importance of SEL competencies, education stakeholders require 
additional guidance on how to use this kind of survey data to inform and improve their programming and 
interventions. Because of this need, as well as their ambitious goals and priorities, DCPS partnered with Regional 
Educational Laboratory (REL) Mid-Atlantic in examining how their existing data might be used to inform education 
decisions and improve key student outcomes, including those highlighted in their 2022 goals. By exploring avenues 
to identify students who could benefit from additional support—such as those who report not feeling loved, 
challenged, and prepared—the findings may suggest how DCPS can best target its growing investments in SEL and 
school climate. Because improvement practices differ across schools, a better understanding of its SEL and school 
climate data would allow the district to enhance the design, consistency, and targeting of its practices and 
programming to attain its goals. Furthermore, DCPS might better identify and serve students at risk of poor 
outcomes and improve the quality and ratings of low-performing schools. 


Box 1. Key terms 


Academic measures. The study used academic measures that serve as both predictors and student outcomes, which include 
academic proficiencies (such as proficiency/college readiness in math and English language arts) and academic behaviors 
(such as attendance, suspensions, grade progression, and credits earned toward graduation) (see table B3 in appendix B). 


Classification accuracy. The percentage of students whose outcomes are correctly classified by a statistical model. 


Demographic variables. The study used demographic data, which include students’ gender, special education status, English 
learner status, grade level, and race/ethnicity, as well as the District of Columbia Public Schools’ (DCPS) classification of 
whether the student is at risk (see table B3 in appendix B). 


Predictive power. The strength of association between a predictor or group of predictors and student outcomes. 


School climate scales. DCPS uses a customized version of Panorama Education’s survey of school climate and social and 
emotional learning (SEL) competencies. The school climate scales include items that ask students, parents, and teachers about 
their personal experiences at the school (see table B1 in appendix B). When referring to individual respondents’ reports, this 
study uses the term school experiences. Averaging the measures of school experiences across respondents at a school 
provides a measure of a school’s climate. Response options are Likert scales relating to the question text (such as agreement 
or frequency), with answers of (1) indicating a low level and (5) indicating a high level. The survey covers three components 
of school experiences: 
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e Rigorous expectations: A 1- to 5-point scale based on items designed to capture “how much students feel that their 
teachers hold them to high expectations around effort, understanding, persistence, and performance in class.” @ 

e Sense of belonging: A 1- to 5-point scale based on items designed to capture “how much students feel that they are valued 
members of the school community.” @ 

e Student satisfaction: A 1- to 5-point scale based on items developed by DCPS to capture student reports of how satisfied 
they are with their school experience. ° 


SEL competency scales. The survey also covers four SEL competencies: 


e Perseverance (also called grit): A 1- to 5-point scale based on items designed to capture “how well students are able to 
persevere through setbacks to achieve important long-term goals (not limited to academics), taking into account their 
experiences and identities.” ? 

e Self-management: A 1- to 5-point point scale based on items for all students designed to capture “how well students 
manage their emotions, thoughts, and behaviors in different situations.” @ 

e Self-efficacy: A 1- to 5-point scale based on items designed to capture “how much students believe they can succeed in 
achieving academic outcomes.” @ 

e Social awareness: A 1- to 5-point scale based on items designed to capture “how well students consider the perspectives 


of others and empathize with them.” @ 


Student Loved, Challenged, and Prepared Index. The study used a student outcome developed to measure progress toward 
DCPS’s strategic goal of having 100 percent of students feeling loved, challenged, and prepared. The Index indicates whether 
or not students feel loved, challenged, and prepared and is based on student reports of perseverance, self-management, self- 
efficacy, sense of belonging, and rigorous expectations (see table B3 in appendix B; see background on the development of 
the Index in appendix A). 


Note 


a. Panorama Education, n.d. 


b. The student satisfaction items have four response categories. When reporting findings, the study team rescaled the student satisfaction scale to range 
from 1 to 5, so it was comparable to the other scales. See appendix B for details on the rescaling. 


Research questions 


This study addresses four key research questions that will help DCPS understand and use measures of SEL 
competencies and school experiences. 


Research question 1. How do average SEL competencies and school experiences differ across grade levels and 
change for individual students between years? Do student and teacher reports of SEL competencies and school 
experiences change in similar ways across grade levels? To what extent do the average differences in students’ 
SEL competencies and school experiences across grades differ by the type of students (such as students classified 
by gender, race/ethnicity, and academic achievement)? How are individual students’ reports of SEL competencies 
and school experiences associated between years, and how does that association compare to that of other 
variables (such as achievement test scores, absences, and suspensions)? 


To identify and support students who need the most help, DCPS requires a greater understanding of whether 
students enrolled in some grades tend to have lower SEL competencies and have poorer school experiences and 
how much students’ reports of SEL competencies and school experiences change between years. Evidence from 
other districts suggests that students’ self-reported SEL competencies do not necessarily improve—and can even 
decline—across grades (West et al., 2018). If DCPS’s data reveal a similar decline, then DCPS might focus its SEL 
efforts on grades before or during the times when students tend to struggle most. Similarly, if students in some 
grades have worse school experiences, DCPS might focus school climate improvement efforts on those grades. 
Finally, if some subgroups of students struggle more than others, DCPS could target supports to them. Information 
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on how individual students’ SEL competencies and school experiences relate between successive years provides 
a sense of the stability of the measures over time. 


Research question 2. To what extent do year-to-year changes in individual students’ SEL competencies and 
school experiences differ across schools? 


The extent to which year-to-year changes in students’ SEL competencies and school experiences differ across 
schools can provide a basis for exploring differences in practices across schools and shed light on the promise of 
targeting supports to specific schools. If year-to-year changes vary across schools, then students in some schools 
have, on average, relatively more positive changes in SEL competencies and school experiences than students in 
other schools. In this case, DCPS may wish to explore the SEL- and climate-related practices of all schools, 
observing whether some practices are consistently associated with more positive outcomes than others. At the 
same time, other factors could matter as well. For example, if family or community supports tend to be greater at 
some schools than others, then school practices might not account for the systematic differences in year-to-year 
changes in SEL competencies and school experiences across schools. Regardless of the source of any differences, 
schools with little or no positive change might benefit from additional supports to help improve students’ growth 
in SEL competencies and school experiences. 


Research question 3. How do measures of SEL competencies and school experiences relate to future outcomes, 
and how do they complement other available data for predicting future outcomes? To what extent do individual 
SEL competencies and school experiences relate to student outcomes measured one and two years later (such as 
achievement test scores, absences, suspensions, and whether a student feels loved, challenged, and prepared)? 
When other data are available—such as demographic information, achievement test scores, absences, and 
suspensions—to predict students’ future outcomes, to what extent does adding measures of SEL competencies 
and school experiences improve the predictive power and accuracy of those predictions? Which types of data and 
statistical models could best help DCPS classify whether students are at risk of having negative outcomes? 


Information about the relationships between current SEL competencies and school experiences and future 
outcomes can help DCPS prioritize which SEL competencies and aspects of school climate to focus on, as well as 
refine ways to identify students at risk of poor future outcomes. For many interventions and initiatives, the 
primary goal is to improve students’ SEL competencies and school experiences in ways that also boost meaningful 
longer-term outcomes, such as grade progression and high school graduation. Similarly, for identifying struggling 
students, a primary goal is to use proximate data to identify students at risk of performing poorly on later 
outcomes, like dropping out of school. Research question 3 explores the predictive power of the SEL competencies 
and school experiences and compares their predictive power to that of other administrative data. 


These analyses will inform DCPS in two distinct ways. First, they will suggest which SEL competencies and student 
experiences DCPS might target to improve longer-term student outcomes. Given that DCPS is already investing in 
this area, the results will inform which of the competencies and school experiences to prioritize. Second, they will 
suggest how DCPS can best identify students who might be at risk of falling behind in the future and whether data 
on SEL competencies and school experiences can improve these predictions. These issues are separate, because 
it is possible that measures of SEL competencies and school experiences relate to future outcomes but they would 
not improve the ability to predict outcomes beyond using other available data. The comparison of different 
statistical models will also provide practical guidance to DCPS on the benefits of different approaches to 
prediction. 
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Subgroup analyses of grades 3 and 8 will inform specific teams at DCPS.* Analyses of grade 3 students will suggest 
to DCPS’s early literacy team which SEL competencies and school experiences in elementary school are most 
critical for early English language arts (ELA) achievement. Analyses of grade 8 students will help DCPS’s graduation 
team to understand which SEL competencies and school experiences in middle school enable students to 
successfully transition to high school, a particularly challenging transition for many (Benner, 2011). 


Research question 4. How do measures of perseverance and rigorous expectations align across students, 
parents, and teachers? Across schools, to what extent do survey reports on these measures from students, 
parents, and teachers align? Is alignment associated with characteristics of schools (such as the demographic 
characteristics of their student population and the schools’ accountability ratings) and response rates on the 
survey? 


Information on the extent to which respondents’ reports align on these two measures will allow DCPS to better 
understand how to address shortfalls in students’ SEL competencies and school experiences and improve 
communication among students, parents, and teachers. After reviewing the 2017/18 survey data, school staff 
struggled to understand why some students reported low levels of SEL competencies and school experiences. If 
reports across respondents are misaligned, it might help the district pinpoint the cause of these gaps, particularly 
around communication and engagement with available programming. For example, if teachers at a school indicate 
that students are prepared but the students report that they do not feel prepared, then school leaders might take 
a different action than if teachers and students agreed that the students are not prepared. Research question 4 
addresses this by examining alignment between school-level measures of SEL competencies and school 
experiences across the three respondent types. The results may also help DCPS interpret results from school-level 
tracking and determine whether including information from different types of respondents provides different 
information. 


The data sources, sample, and methods used to answer these questions are described in box 2 and appendix B. 


Box 2. Data sources, sample, and methods 


Data sources. The key data sources for this study are DCPS’s Panorama Education student, teacher, and parent surveys and 
administrative records from the 2017/18, 2018/19, and 2019/20 school years (see, for example, DCPS, 2019). The COVID-19 
pandemic affected the collection of the survey data from the 2019/20 school year (see appendix B), which limited its use in 
this study. Our main analyses, therefore, did not include the 2019/20 survey data. The administrative data covered a range 
of student demographic characteristics and academic measures, as well as school characteristics (see appendix B). 


Sample. During the 2017/18-2018/19 school years, there were 39,791 unique DCPS students enrolled in grades 3 through 12 
(see table B2 in appendix B). Of these students, 30,462 responded to the Panorama Education survey in one or more years 
and could therefore be included in the analyses. The study sample included 4,273 unique teacher respondents and 12,216 
parent responses on the SEL surveys during the 2017/18-—2018/19 school years (no 2019/20 data were used in these 
analyses). Sample sizes and number of observations for each analysis are in table B4. 


Methodology. 


Nonresponse analyses. The study team assessed survey nonresponse bias in the student and teacher surveys (see appendix 
B). The results suggested the potential for nonresponse bias, which could lead to findings that do not represent the student 
and teacher populations across DCPS. As a result, the analyses of student and teacher data used nonresponse weights based 
on administrative data that were available for students and teachers regardless of whether they completed the survey. Due 
to a lack of data on parents who did not respond, the study team did not assess nonresponse bias on the parent survey. 


Research question 1. To inform how measures of SEL competencies and school experiences differ across grades, the study 


team calculated the average level of each measure reported by students and teachers by grade levels within the two academic 


1 To provide additional context, the study team also explored the link between credits behind in grade 9 and high school graduation, 
discussed in appendices B and C. 
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years included in the sample. The student reports were averaged within individual grades. Because a teacher’s report could 
apply to students in multiple grades, the teacher reports were averaged within ranges of grades based on the level of their 
school: grades K—5 (elementary), grades 6-8 (middle), or grades 9-12 (high). To assess whether the measures varied across 
grades and school levels, the study team conducted F-tests of the null hypothesis that the measures were equal across grades 
and school levels. These analyses compared students in different grades when the survey was administered. To understand 
how individual students developed between years, the study team estimated the correlation between student reports of the 
same measure over time (the year-to-year correlation) and between other variables like test scores, absences, suspensions. 


Research question 2. The study team examined the extent to which changes in students’ SEL competencies and school 
experiences differed across schools. For each measure, the study team calculated the expected difference in percentile points 
between students who attend a school with a high level of (positive) change compared to a school with an average level of 
change. A high level of change is defined as one standard deviation above average within DCPS. As described in appendix B, 
the calculation was based on the intraclass correlation coefficient (ICC) of year-to-year changes in students’ measures. The 
ICC represents the fraction of a measure’s total variance that arises across schools compared to within schools. A higher 
percentile point difference (and ICC) indicates that changes in students’ measures vary more across schools. As a benchmark, 
the analyses also examined changes in achievement test scores, suspensions, and in-seat attendance, as well as the variation 
across schools for measures during a fixed year (as opposed to changes across years). 


Research question 3. To describe how individual measures of SEL competencies and school experiences relate to later 
outcomes, the study team estimated pairwise correlations between each measure and outcome. If the pairwise correlation 
is positive, then, on average, when one variable takes a higher value, the other variable takes a higher value. Similarly, 
multivariate correlations (the square root of the adjusted R-squared statistic) provided evidence on the extent to which 
groups of measures predicted student outcomes. The higher the multivariate correlation, the more related the variable is to 
the group of other variables. This correlation also suggests how predictions would improve when variables are added to a 
group of predictors. If adding a variable to a group of predictors increases the multivariate correlation, then the variable 
provides additional predictive power above and beyond the initial group. The groups included students’ SEL competencies 
and school experiences, demographic variables, academic measures (such as in-seat attendance), and all three types of 
predictors combined. By comparing the results from these groups of predictors, the study team assessed the extent to which 
the measures of SEL competencies and school experiences added predictive power above and beyond the academic and 
demographic variables that DCPS previously collected. 


The study team conducted additional analyses to provide information on how well DCPS could use different data sources to 
identify whether or not individual students are at risk of poor outcomes. The study team recoded the continuous outcomes 
used in the correlational analyses to be dichotomous outcomes (for example, whether or not a student was chronically 
absent). Probit and machine learning models (random forests) were used to calculate how frequently different groups of 
predictors classified students’ future outcomes accurately. These analyses complement the correlational analyses by placing 
the results in terms of the accuracy of identifying students who are at risk of having negative future outcomes. 


Research question 4. Finally, the study team conducted two analyses to examine the alignment among students, teachers, 
and parents in their reports of perseverance (an SEL competency) and rigorous expectations (a school experience)—the two 
measures available in all three surveys. Both analyses involved examining school-level averages of the reports from the three 
types of respondents, as teacher and parent responses cannot be linked to individual students. First, the study team 
estimated pairwise correlations between the school-level averages among the respondent types. Second, the study team 
compared the averages between school-level reports across respondent types. 


Due to a lack of data on parents who did not respond to the survey, these analyses cannot account for nonresponse in parent 
reports (see appendix B). For consistency, the study team did not use nonresponse weights when calculating the school-level 
averages of student and teacher results. The results, therefore, generalize to the population of students, teachers, and 
parents who responded to the survey, as opposed to the full population. 


2 Supplemental analyses reveal that adjusting for nonresponse in the student and teacher reports made little difference, suggesting that 
the results for these two groups may generalize to the full population. However, given the low estimated response rate among parents— 
approximately 12 and 15 percent in 2017/18 and 2018/19, respectively (DCPS, 2019)—the study team strongly cautions against generalizing 
the analyses that include parent reports. 
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Benchmarks. This study uses two benchmarks when describing the strength of correlations and differences in SEL 
competencies and school experiences between groups of students. 

1. Correlations. Based on past evidence on the extent to which cognitive (IQ and achievement) tests predict other 
academic outcomes, this study describes ranges of correlations as follows: 0.0 to 0.09 is low; 0.10 to 0.19 is moderate; 
0.20 to 0.29 is substantive; and 0.30 and above is high (see appendix B for details). These descriptions also apply when 
describing the difference between two correlations. 

2. Comparing SEL competencies and school experiences between students. Based on past evidence on the extent to 
which school-based SEL programs improve students’ SEL competencies and school experiences, this study describes 
differences in SEL competencies and school experiences between groups of students as follows: 0.0 to 0.09 standard 
deviations is small; 0.10 to 0.19 standard deviations is moderate; 0.20 to 0.29 standard deviations is substantive; and 
0.30 standard deviations and above is large (see appendix B for details). 


More information about the study data sources, sample, and methods is in appendix B. 


Findings 


This section presents findings to address the study’s research questions, with additional findings in appendix C. 


Both student and teacher reports of SEL competencies and school experiences were highest in 
elementary school and lowest in middle school and high school 

Student reports of SEL competencies and school experiences exhibited a U-shaped pattern, with the lowest 
reports in middle school and the beginning of high school 


For all student-reported SEL competencies and school experiences, the average value was highest in elementary 
school (grades 3-5) and dipped in middle school (grades 6-8) and the first half of high school (grades 9 and 10), 
at which point the average value began to rise again (figure 1).° In addition, formal tests indicated that each 
measure varied across grades in a statistically significant way. Of all of the measures, rigorous expectations 
differed the most across grades, with a range of 0.71 on the 5-point scale—representing a large difference in terms 
of the study’s benchmarks.* Of the SEL competencies, self-efficacy showed the biggest differences across grades. 
The general patterns are also consistent with those from the CORE Districts, a network of large urban districts in 
California (West et al., 2020).° Students’ average SEL competencies and school experiences also exhibited similar 
patterns within student subgroups as they do in the full sample (see tables C1—-C7 in appendix C). Notably, in nearly 
every grade level, female students reported moderately higher levels of self-management and social awareness 
than their male peers (differences that were statistically significant). 


3 The year-to-year correlations between individual SEL competencies and school experiences tended to be lower than those of academic 
measures, consistent with the findings that the SEL competencies and school experiences evolve substantially across grades (see figure C2). 
4 The patterns were similar when estimating averages by age (see figure C1 in appendix C) and for subgroups of students (see tables C1-C7). 
5 Importantly, these estimates should not be interpreted as how measures of individual students’ SEL competencies and school experiences 
change over time, because they involve comparing different students across cohorts and grades. The grade-level patterns in measures of 
SEL competencies and school experiences were nearly identical between the two survey years in the study despite coming from different 
cohorts, suggesting that differences across cohorts played a minimal role (see tables C1—C7 in appendix C). However, the study cannot rule 
out that the U-shaped pattern emerged due to changes in the composition of enrolled students over time. 
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Figure 1. Student self-reports of SEL competencies and school experiences peaked in elementary school, 
declined in middle school and early high school, and increased at the end of high school 
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SEL is social and emotional learning. 

Note: The figure shows for each grade the mean of each SEL competency and school climate scale (described in appendix B) for the 2017/18 and 2018/19 
school years combined. The means were calculated using nonresponse weights, as described in appendix B. F-tests of the null hypothesis that the scales 
were equal across grades are significant at p <.001 for each scale. 

Source: Authors’ analyses based on survey and administrative data provided by the District of Columbia Public Schools, 2017/18 to 2018/19. 


For the teacher-reported measures of perseverance and rigorous expectations, the average values were highest 
in elementary school (grades K—-5) and declined through middle school (grades 6-8) and high school (grades 9-12) 
(represented by the dotted lines in figure 2). In addition, formal tests indicated that each measure varied across 
school levels in a statistically significant way. Although the teacher reports did not exhibit a U-shaped pattern, 
they were consistent with the student reports. Because the teacher data were averaged within each school level— 
that is, elementary, middle, and high school grades—declines within individual grades might be offset by increases 
in other grades. Supporting this possibility, averaging the student reports to the school level (rather than grade 
level) produced a similar pattern as the teacher reports, because the student reports of these two measures were 
lowest in grades 9 and 10 but higher in grades 11 and 12 (represented by the solid lines in figure 2). The steepness 
of the decline differed between the teacher and student reports, with the decline in perseverance being steepest 
for teacher reports and the decline in rigorous expectations being steepest for student reports. 


Figure 2. School-level teacher and student reports of students’ SEL competencies and school experiences 
declined from elementary school to high school 
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SEL is social and emotional learning. 

Note: The figure shows, for each level of school, the mean of each SEL competency and school climate scale (described in appendix B) for the 2017/18 and 
2018/19 school years combined. The means were calculated using nonresponse weights, as described in appendix B. F-tests of the null hypothesis that the 
scales were equal across school levels are significant at p <.001 for each scale. 


Source: Authors’ analyses based on survey and administrative data provided by the District of Columbia Public Schools, 2017/18 to 2018/19. 


REL 2021-114 8 


Year-to-year changes in students’ SEL competencies and school experiences differed across schools, 
and toa similar degree as year-to-year changes in academic measures 

Among schools with high levels of positive change (one standard deviation above average), the average year-to- 
year change in students’ SEL competencies and school experiences tended to be 5.9 to 8.3 percentile points higher 
than among schools with average year-to-year change (figure 3). This was less than those for changes in in-seat 
attendance, similar to those for suspensions, and somewhat higher than those for math and ELA achievement test 
scores.” ’ These findings indicate that changes in SEL competencies and school experiences differed across schools 
to a similar degree as changes in academic measures.® ° 


Figure 3. Differences across schools in year-to-year changes in SEL competencies and school experiences were 
similar to those for academic measures 
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Self-management 
Self-efficacy ELA achievement 
Social awareness 
Rigorous expectations lnseat allendance 

Sense of belonging 

Student satisfaction 8.03 

012 3 4 5 6 7 8 9 10 11 12 13 012 3 4 5 6 7 8 9 10 11 12 13 
Difference in percentiles Difference in percentiles 


i] Academic measures | School experiences = SEL competencies 


ELA is English language arts. SEL is social and emotional learning. 

Note: The figure shows the difference in percentiles of each measure between schools with average year-to-year improvements and schools with high year- 
to-year improvements (one standard deviation above average). The measures are defined in table B3 in appendix B. 

Source: Authors’ analyses based on survey and administrative data provided by the District of Columbia Public Schools, 2017/18 to 2018/19. 


SEL competencies and school experiences were related to students’ future outcomes, but their 
predictive power relative to academic measures varied 


Of the SEL competencies and school experiences, self-management was most strongly associated with students’ 
future outcomes 


Self-management was the SEL competency or school experience most strongly correlated with six of the nine 
outcomes included in the analysis, with correlations ranging from 0.09 to 0.23 (which the study considers a low- 
to-substantive range; see panel a of table 1).1° +1 For the grade progression and in-seat attendance outcomes, the 
correlations were strongest (at the moderate range) for rigorous expectations, but other SEL competencies and 
school experiences were similarly correlated with these outcomes. Finally, whether students felt loved, 
challenged, and prepared was the most related to sense of belonging—this was likely because whether students 


6 The patterns are similar when considering the ICCs, rather than percentile measures (see figure C3 in appendix C). 

7 The estimates of the ICCs for year-to-year changes in math and ELA achievement test scores are comparable to those compiled in Schochet 
& Chiang (2010). 

8 Compared to the year-to-year changes in measures, the measures during a fixed year tended to vary more across schools (see figure C4 
in appendix C), consistent with existing literature that has examined achievement test scores (Schochet & Chiang, 2010). 

° The difference in year-to-year changes between average- and high-change schools was substantial relative to the year-to-year change in 
average schools (see table C9 in appendix C). 

10 The outcomes were all recoded so that a higher value of the outcome is beneficial. For example, a positive correlation between a predictor 
and the number of suspensions indicates that higher values of the predictor are associated with fewer suspensions. 

1 The results were similar when examining the predictive power of outcomes measured one year later (see table C13 in appendix C). 
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felt loved is based on items from the sense of belonging scale.** With a few exceptions, the remaining measures 
of SEL competencies and school experiences were also positively related to students’ future outcomes at lower 
levels. 


Findings for key subgroups of students of interest to DCPS were consistent with findings from the main sample: 


e Early literacy. The findings were similar when considering relationships between grade-3 SEL competencies and 
school experiences and grade-3 ELA achievement, with the strength of the correlations ranging from low to 
substantive (see table C10 in appendix C). As in the main sample, self-management was the most correlated 
with grade-3 ELA achievement. 


e Grade 8 predictors of high school success. Students’ SEL competencies and school experiences in grade 8 were 
moderately predictive of the number of credits they were behind at the end of grade 9 (see table C11 in 
appendix C). These grade-8 SEL competencies and school experiences are likely related to students’ eventual 
graduation, because the extent to which students are behind in credits in grade 9, in turn, accurately classifies 
whether students will graduate 75 percent of the time (see table C12). 


Of the individual predictors, the best predictor of a later student outcome tended to be an earlier measure of that 
outcome 


Compared to the SEL competencies and school experiences, early academic measures tended to have stronger 
relationships with future academic outcomes (panel b of table 1). For example, of all predictors, ELA achievement 
in 2017/18 was the most correlated with ELA achievement in 2019/20, with a high correlation of 0.78. Similarly, 
for the one outcome based on SEL competencies and school experiences—whether students felt loved, 
challenged, and prepared—the best individual predictors were SEL competencies and school experiences. Grade 
progression, graduation, Advanced Placement (AP) credits earned, and credits behind did not have earlier 
measures based on those outcomes. Of those, in-seat attendance was most strongly related to future grade 
progression, credits behind, and graduation. ELA achievement was the most strongly related to AP credits earned. 
For all academic outcomes, the strongest individual correlation was with an academic measure, rather than a 
measure of an SEL competency or school experience (compare panels a and b in table 1). 


As a group, SEL competencies and school experiences did not help predict future academic outcomes more 
accurately when demographic and academic measures were available, but they did help predict whether 
students felt loved, challenged, and prepared 


As a group, demographic and academic measures strongly predicted future academic outcomes, and adding 
information on SEL competencies and school experiences did not improve those predictions. Compared to the 
group of SEL competencies and school experiences, the group of demographics and academic measures better 
predicted academic outcomes, with high multivariate correlations ranging from 0.33 to 0.87 (see panel d in table 
1). For academic outcomes, when using all predictors, the correlation was, at most, 0.01 higher than when using 
only the demographic and academic measures (compare the last two rows in panel d). Although the SEL 
competencies and school experiences were moderately to highly predictive of academic outcomes on their own, 
they added little predictive power because the information they capture was also captured by the demographic 
and academic predictors. In contrast, adding the SEL competencies and school experiences improved the 
prediction of whether students felt loved, challenged, and prepared (compare the last two rows in panel d). 


12 In supplementary analyses, the study team examined the correlations between school-level teacher and parent reports of perseverance 
and rigorous expectations and school-level outcomes one year later (see table C14 in appendix C). Both perseverance and rigorous 
expectations were correlated with outcomes, but the relative strength of the correlations differed by respondent type. The teacher reports 
of perseverance were more positively correlated with outcomes than were the reports of rigorous expectations. However, the reverse was 
true of the parent reports for academic behaviors (suspensions and attendance) and whether students felt loved, challenged, and prepared. 
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Table 1. Correlations between predictors and students’ outcomes measured two years out 


Outcomes measured two years out in 2019/20 


Progressed successfully 
Number of suspensions 
Math achievement 

ELA achievement 
In-seat attendance 
Number of AP credits 
earned (grades 10-12) 
Number of credits 
behind (grades 3-12) 
(efe-lol Ue} acom Tidal ia W4 
Loved, challenged, and 
prepared (grades 3-12) 


Predictor(s) in 2017/18 
a. SEL competencies and school experiences 


Perseverance 0.06 0.01 -0.05 -0.05 0.09 -0.04 0.03 0.07 0.22 
Self-management 0.09 0.11 0.23 0.23 0.14 0.18 0.12 0.16 0.21 
Self-efficacy 0.10 0.01 0.08 0.09 0.14 0.07 0.05 0.06 0.25 
Social awareness 0.10 0.07 0.13 0.13 0.17 0.14 0.06 0.09 0.25 
Rigorous expectations 0.12 0.02 0.04 0.04 0.17 0.02 0.06 0.10 0.21 
Sense of belonging 0.08 0.01 0.00 0.00 0.13 0.03 0.00 0.04 0.29 
Student satisfaction 0.09 0.06 0.11 0.11 0.17 0.10 0.05 -0.02 0.24 


b. Academic measures 
Math achievement 
ELA achievement 0.23 0.15 
In-seat attendance 


0.28 0.24 0.01 


Number of suspensions 


c. Groups of variables 

SEL competencies and school 
experiences 
Demographics 


Academic measures 
d. Combinations of groups of predictors 


Demographics and academic 


measures 
Absolute value of the 
correlation Low Moderate Substantive 


All predictors 
0.00-0.09 0.10-0.19 0.20-0.29 0.30+ 


AP is Advanced Placement. ELA is English language arts. SEL is social and emotional learning. 

Note: The table shows pairwise correlations and multivariate correlations between predictors in the left column and the outcomes in the top row. The 
outcomes were all recoded so that a higher value of the outcome is beneficial. The bold font indicates the correlation with the highest absolute value within 
the column and panel. The correlations were calculated using nonresponse weights, as described in appendix B. The sample included students who 
completed the SEL survey in 2017/18. The math and ELA achievement outcomes were based on high school students’ Preliminary SAT (PSAT) and SAT scores. 
Source: Authors’ analyses based on survey and administrative data provided by the District of Columbia Public Schools, 2017/18 to 2019/20. 


Overall, SEL competencies and school experiences added little value in classifying whether students were at risk 
of poor academic outcomes beyond demographic and academic predictors 


Across all outcomes, the predictive models that used demographic characteristics and academic measures 
accurately classified students’ outcomes between 63.1 and 95.3 percent of the time (figure 4).‘* Consistent with 
the multivariate correlations, the classification accuracy for academic outcomes improved by at most 0.6 
percentage points when adding the SEL competencies and school experience variables to models that included 
demographic characteristics and academic measures (see table C15 in appendix C). However, adding the SEL 
competencies and school experiences increased the accuracy by up to 5.2 percentage points when classifying 


13 Complex machine learning algorithms did not systematically perform better compared to probit models when assumptions and modeling 
approaches were aligned (see table C15 in appendix C). 
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whether students felt loved, challenged, and prepared two years later—a measure based on future SEL 
competencies and school experiences. Relative to the accuracy of classifying individual students’ outcomes using 
the typical outcome observed among their peers (that is, the null model), the predictive models improved the 
classification accuracy for some outcomes more than others (see table C15).’* For example, the predictive model 
improved the classification accuracy of college readiness in ELA two years later by 27.8 percentage points. On the 
other hand, the predictive model improved the classification accuracy of whether students would be suspended 
two years later by only 0.1 percentage points. Notably, the overall classification accuracy does not distinguish 
between students who have negative versus positive outcomes. If DCPS places more importance on classifying 
students who eventually have negative outcomes, DCPS could refine the models to do so (box 3). Additional 
evidence suggests that these general findings hold for refinements that place more weight on identifying students 
who eventually have negative outcomes (see figure C5). 


Figure 4. Adding SEL competency and school experience predictors improved classification accuracy by at 
most 5.2 percentage points relative to models that included demographic and academic predictors 


Not college ready 
in ELA 


Did not progress , 
successfully 95.3% Did not graduate 88.6% 


Red ACGR status 


Not college ready 90.6% Not loved, challenged 68.3% 
in math ide and prepared : 


0 25 50 75 100 0 25 50 75 100 


Classification accuracy (percentage) Classification accuracy (percentage) 


; Improvement with SEL competencies 
fed Demographic and academic predictors ea and school experiences 


ACGR is Adjusted Cohort Graduation Rate. AP is Advanced Placement. ELA is English language arts. 

Note: The figure shows the accuracy of predictive models that used demographic variables and academic measures to predict outcomes two years later, as 
well as the improvement in accuracy by adding SEL competencies and school experiences (described in appendix B). The estimates were based on probit 
models and calculated using nonresponse weights, as described in appendix B. The measures are defined in table B3. 

Source: Authors’ analyses based on survey and administrative data provided by the District of Columbia Public Schools, 2017/18 to 2019/20. 


Box 3. Interpreting classification accuracy 


This study estimated how well predictive models accurately classified the future outcomes of individual students. For each 
outcome, the model generated the probability that each student will have a negative outcome based on a set of predictors. 
Each student with a probability above a threshold was classified as at risk of having a negative outcome (for example, not 
progressing to the next grade), and those below the threshold were classified as not at risk of a negative outcome. The study 
selected the thresholds to maximize the overall classification accuracy—the percentage of individual students who were 
classified correctly, regardless of whether they eventually had positive or negative outcomes. The overall classification 
accuracy can be split into two components: 


1. The percentage of students with a negative outcome whom the model accurately classified. This component measures 
how accurately the model and threshold classified individual students who eventually had a negative outcome (for 
example, did not graduate). 

2. The percentage of students with a positive outcome whom the model accurately classified. This component measures 
how accurately the model and threshold classified individual students who eventually had a positive outcome (for 
example, did graduate). 


M4 The baseline accuracy is the percentage of students who would be correctly classified if all students were classified with the most 
prevalent value for each outcome. For example, because 34.9 percent of students were chronically absent two years out, the baseline 
accuracy was 65.1 percent—the classification accuracy from assuming that all students were not chronically absent. For additional details, 
see appendix B. 
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The probability threshold governs a tradeoff between accurately classifying students who eventually have negative and 
positive outcomes. Reducing the threshold means that students who have a positive outcome are accurately classified a 
higher percentage of the time, but students with a negative outcome are accurately classified a lower percentage of the time. 
For some outcomes, it might be more important to accurately classify students who eventually have negative outcomes to 
ensure resources are allocated to students most likely to require supports, even at the expense of misclassifying some 
students who eventually have positive outcomes. Refining the threshold can give more weight to such students. A receiver 
operating characteristic curve illustrates this tradeoff for all possible thresholds and can be used as a tool for determining 
such refinements (see figure C5 in appendix C). 


At the school level, student, parent, and teacher reports of perseverance and rigorous expectations 
were positively related; however, teacher and parent reports exhibited the greatest differences 

When comparing student, parent, and teacher reports of perseverance and rigorous expectations, schools with 
more favorable reports from one group tended to, but did not always, have more favorable reports from the other 
groups.’ The correlation between school-level averages and respondent reports of perseverance and rigorous 
expectations ranged from 0.09 (low) to 0.59 (high; figure 5). This range is comparable to estimates from other 
studies that have explored the alignment of student, parent, and teacher reports of individual students’ SEL 
competencies (Barbaranelli et al., 2003; Elliott et al., 2020). The correlations among respondent types for rigorous 
expectations were substantially higher than those for perseverance, suggesting more alignment. 


For both rigorous expectations and perseverance, the correlations between parents and teacher reports were 
lower than any other pairs—that is, teacher-student and parent-student—suggesting the least alignment between 
parents and teachers. These basic correlations indicate how measures of each respondent type move together. 
For example, they showed that schools with higher student reports tended to have higher teacher reports. 
However, they did not inform whether the responses align on average. 


Figure 5. Of the pairs of respondent reports, teacher and parent reports were the least correlated across 
schools 


(a) Perseverance (b) Rigorous expectations 

2 Teacher & student Ed 2 Teacher & student} # 
a a 
r= 1-4 
Student & parent 0.17 8S Student & parent} 0.59*** Fa 
5 5 
Qa Qa 
no no 
(Teacher & parent | 009] (© Teacher & parent} 0.24** 

0.0 01 02 03 04 05 06 0.0 01 02 03 04 05 06 

Correlation Correlation 


* Significant at p < .05; *** significant at p < .001. 

# Correlation differed from scale-specific “teacher & parent” (the lowest) correlation by 0.10 or more—the cutoff for a moderate difference between 
correlations. 

Note: The figure shows the pairwise correlation of school-level averages of the scales (described in appendix B) between each pair of respondents. 

Source: Authors’ analyses based on survey and administrative data provided by the District of Columbia Public Schools, 2017/18 to 2018/19. 


Teachers who responded to the survey provided less favorable ratings of students’ perseverance and more 
favorable ratings of rigorous expectations than did students or parents (figure 6). On average, teacher reports of 
perseverance were lower than student and parent reports by 0.66 and 0.54 points on a 1- to 5-point scale, 
respectively. On the other hand, teacher reports of rigorous expectations were higher than student and parent 
reports by 0.37 and 0.42 points on a 1- to 5-point scale, respectively. Although the student and teacher reports 
were correlated, this finding indicates that teachers consistently reported different average levels compared to 
students. The average school-level differences between respondent types were related to the characteristics of 


15 The response rates for the parent survey were substantially lower than the other two, so caution should be used when considering the 
parent findings. 
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the schools (including School Transparency and Reporting [STAR] rating and geographic ward), suggesting that 
alignment varied systematically across schools (see tables C16—C19 in appendix C). 


Although it is not possible to explore nonresponse bias for parents, the study team conducted analyses using 
student and teacher nonresponse weights. Relative to the analyses without weights, the correlations between 
respondent types changed by at most 0.02 (see figure C6 in appendix C), and the average differences between 
respondent types changed by at most 0.01 (see figure C7). These estimates suggest that nonresponse bias on the 
student and teacher surveys likely did not drive the results. However, the study team cannot rule out that 
nonresponse bias affected the parent survey, so it suggests caution when interpreting those findings. 


Figure 6. Average reports on perseverance and rigorous expectations differed across respondent types mainly 
because teachers responded differently than students and parents 


(a) Perseverance (b) Rigorous expectations 
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# Absolute value of difference relative to students met or exceeded 0.10. 
Note: The figure shows the average school-level reports of the scales (described in appendix B) for parents, teachers, and students. 
Source: Authors’ analyses based on survey and administrative data provided by the District of Columbia Public Schools, 2017/18 to 2018/19. 


Limitations 


Readers should keep one primary limitation in mind as they consider this study’s findings: it examined 
correlational relationships between variables, not causal ones (see appendix B for other limitations). The results 
suggest which SEL competencies and school experiences were associated with student outcomes. Although it is 
possible that SEL competencies and school experiences could cause those outcomes, it is also possible that the 
variables were associated because other factors caused better student outcomes as well as better SEL 
competencies and school experiences. For example, supportive parents could both foster students’ SEL 
competencies and improve students’ outcomes by assisting them with schoolwork. Therefore, the findings do not 
imply that improvements in SEL competencies or school experiences would necessarily improve outcomes. 


Implications 

The study addressed several research questions that investigated the properties of measures of SEL competencies 
and school experiences in DCPS. The findings for each research question have implications for DCPS—particularly 
as they relate to the district’s strategic goals (see table Al in appendix A)—and for other districts with access to 
similar types of data. 


Research question 1. The study findings suggest that middle school and early high school could be especially 
important times to support students’ SEL competencies and improve their experiences at school. During middle 
school and early high school, students reported the lowest SEL competencies and school experiences. These 
findings are consistent with other research that has demonstrated that middle school and transitioning to high 
school can be especially challenging (Benner, 2011; Eccles, 2004; Rudolph et al., 2001). Supporting students before 
or during these grades could potentially help DCPS achieve its goal of helping all students to feel loved, challenged, 
and prepared (see table A1 in appendix A). In addition, DCPS could investigate the causes for these declines to 
better understand how to support students. At the same time, the study cannot rule out that this U-shaped 
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pattern emerges because the composition of students changed across grades. For example, if less perseverant 
students dropped out of school after grades 9 and 10, the average perseverance of students who remained 
enrolled in grades 11 and 12 would increase. The results still address the study’s research question, because they 
suggest which grades DCPS could target to improve the SEL competencies and school experiences for enrolled 
students—the students whom DCPS can most easily reach through programming or initiatives in schools. Students 
who have already dropped out would be more challenging to support directly. 


Research question 2. Across schools, year-to-year changes in SEL competencies and school experiences differed 
to a meaningful degree, similar to that of academic measures. These findings could have arisen for at least two 
reasons. First, differences in practices across schools could have led to differences in year-to-year changes in the 
measures. In this case, the findings suggest that schools have the potential to improve SEL competencies and 
school experiences. Second, other factors that were associated with schools could have driven the differences in 
the year-to-year changes. As a next step, DCPS could explore the extent to which practices differ between schools 
that are currently associated with higher growth in SEL competencies and school experiences and those with lower 
levels of growth. If the practices differ systematically, then that would provide additional evidence that school 
practices are driving the differences, and the higher-change schools could potentially serve as a model for lower- 
change schools. These findings also suggest that school-level targeting could be effective. In particular, DCPS’s 
student support teams could work with lower-performing schools to develop plans around any promising SEL- 
related programming. Such efforts could help DCPS achieve its strategic goal that all schools should be highly rated 
or improving (see table A1 in appendix A). 


Research question 3. The study found that students’ SEL competencies and school experiences— especially self- 
management and rigorous expectations—were moderately to substantively related to their later academic 
outcomes; this finding suggests that improving SEL competencies and school experiences may help DCPS progress 
toward its strategic goals around high school graduation, re-enrollment, early literacy, and college and career 
readiness (see table Al in appendix A). These findings are consistent with the broader literature, which has found 
correlations of a similar magnitude and that skills related to self-management tend to be the most related to 
students’ future academic outcomes and longer-term outcomes like job performance (Almlund et al., 2011; 
Poropat, 2009; West et al., 2016). Of the SEL competencies and school experiences, self-management was the 
most related to students’ future outcomes, so DCPS may consider prioritizing strategies designed to promote this 
competency. However, confirming this priority would require additional research, such as an evaluation of 
interventions designed to improve SEL competencies and school experiences. 


At the same time, measures of SEL competencies and school experiences added little value in classifying whether 
students were at risk of poor academic outcomes beyond the other predictors. On their own, the SEL 
competencies and school experiences predicted students’ outcomes. However, they did not add predictive power 
beyond other predictors (demographic variables and prior academic outcomes), because those other predictors 
may have captured information similar to that captured by measures of SEL competencies and school experiences. 
For that reason, DCPS may consider focusing on predictive models that use demographic and academic measures 
as predictors. The measures of SEL competencies and school experiences can still be helpful in supporting 
students. Once at-risk students are identified, their SEL competencies and school experiences may inform how 
schools can support them. In addition, SEL competencies and school experiences improved predictions of whether 
students felt loved, challenged, and prepared, which may be of interest given DCPS’s goal around this outcome. 


Research question 4. DCPS could consider steps to investigate differences between student, teacher, and parent 
reports of perseverance and rigorous expectations. For example, compared to students, teachers provided less 
favorable ratings of students’ perseverance and more favorable ratings of rigorous expectations. DCPS could 
incorporate this type of information in discussions with students and teachers in schools with large apparent 
differences to better understand their perceptions and inform areas for improvement in school quality. 
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Appendix A. About the study 


The District of Columbia Public Schools (DCPS) needed timely information on its social and emotional learning 
(SEL) data in order to meet its 2017—22 Strategic Plan goals. The Strategic Plan is an effort that DCPS uses to hold 
itself accountable to the public. In its 2017 Strategic Plan, DCPS set six concrete and ambitious goals to “[become] 
a district of both excellence and equity” (DCPS, 2017). The Strategic Plan highlights SEL competencies and school 
climate as key components in its strategic priority to “educate the whole child” (DCPS, 2017). All six goals reflect 
this priority, either explicitly or implicitly. For example, DCPS has set an explicit goal that by 2022, 100 percent of 
students will feel “loved, challenged, and prepared,” as measured by an index based on measures of SEL 
competencies and school climate (DCPS, 2017). Other goals—such as improving college and career readiness and 
re-enrollment rates—are implicitly related to SEL competencies in that they could be furthered by better 
identifying at-risk students and boosting SEL competencies. 


DCPS requested the Regional Educational Laboratory (REL) Mid-Atlantic’s support in providing information about 
how to achieve these goals or make substantial progress toward them by 2022. These analyses of SEL and climate 
measures and related administrative data (for example, absences and credits earned) provide DCPS this 
information in order to reach its time-sensitive Strategic Plan goals. 


This study builds on DCPS’s recent work to prioritize and measure SEL competencies and climate perceptions since 
it began administering the Panorama Education survey in spring 2018. It examines the properties of SEL and 
climate measures based on these surveys to inform progress toward DCPS’s six Strategic Plan goals. Supported 
through a 2018/19 REL Mid-Atlantic coaching activity, DCPS has successfully developed and validated measures 
of SEL competencies based on the SEL survey. Each measure captures a different SEL competency (for example, 
perseverance) or aspect of school climate (for example, rigorous expectations) and is based on averaging scores 
of a group of items designed to measure that construct. The student measures span four SEL competencies 
(perseverance, self-management, self-efficacy, and social awareness) and three measures of a school’s climate 
that could boost SEL competencies (rigorous expectations, sense of belonging, and student satisfaction). As part 
of the earlier REL Mid-Atlantic coaching activity, it was demonstrated that the measures met standard criteria for 
reliability and validity (Kautz et al., 2019). In addition, the study team used five of these measures to develop the 
Loved, Challenged, and Prepared Index that DCPS is now using to track and publicly report progress toward its 
ambitious goal that 100 percent of students will feel “loved, challenged, and prepared” by 2022 (figure A1).*© 


16 Using theoretical and empirical evidence, the measures were assigned to each componentas follows: sense of belonging capturing loved; 
rigorous expectations capturing challenged; and perseverance, self-management, and self-efficacy capturing prepared. 
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Figure A1. DCPS’s Loved, Challenged, and Prepared Index reporting 
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Students who feel loved are more likely to attend school, be engaged, and take the risks associated with tackling challenging content. 
DCPS has developed a social emotional and academic development (SEAD) framework that also emphasizes the importance of family 
partnerships in supporting our students. DCPS also provided professional learning for all educators including trauma-informed 
practices and culturally responsive pedagogy, all of which aim to strengthen our schools’ cultures. 


DCPS is District of Columbia Public Schools. 
Source: DCPS, 2018 


While DCPS has made much progress, it still has a long way to go to achieve these goals (table A1). For example, 
to meet Goal 3, DCPS needs to boost its graduation rate by over 15 percentage points, a substantial amount. 


Table A1. How this study addresses each Strategic Plan goal 


Year 1 reported status? How this study could contribute to this goal 
1. Double the percent of students who are e 35.1 percent for proficient e Use DCPS’s data to suggest which SEL 
college and career ready as measured for ELA competencies or aspects of school climate might 
by proficiency on the PARCC. This goal e 30.5 percent proficient for | be most promising to target to boost college and 
represents an increase from 31.9 math career readiness. 


percent to 63.8 percent proficient for 
ELA and 27.4 percent to 54.8 percent 
proficient for math 


2. 100 percent of K-2 students reading on 


65 percent are reading on e Use DCPS’s data to suggest which SEL 


or above grade level or above grade level competencies or aspects of school climate might 
be most promising to target to improve early 
reading skills. 

3. 85 percent of students graduate within e 68.6 percent graduatein © Provide DCPS with new ways to identify students 
four years and 90 percent graduate four years at risk of dropping out or not transitioning 
within five years e 75.5 percent graduate in between grades. 

five years e Use DCPS’s data to suggest which SEL 


competencies or aspects of school climate to 
focus on for individual students. 


4. 100 percent of students feel loved, e 45 percent of students e Provide information to help DCPS identify the 
challenged, and prepared feel loved, challenged, types of students who could benefit from 
and prepared additional support (such as by age, grade, and 


demographic group). 
e Inform whether it might be fruitful to focus on 
particular schools or particular types of schools. 


5. 100 percent of schools highly rated or e Not available e Provide information to help identify which SEL 
improving based on the DC School competencies or aspects of school climate that 
Transparency and Reporting Framework, schools could target to improve school ratings. 


which incorporates test scores, 
advanced classes enrollment rate, 
adjusted cohort graduation rate, in-seat 
attendance rate, re-enrollment 
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(efey-}| Year 1 reported status? How this study could contribute to this goal 
6. 90 percent of students re-enroll and e 84 percent of students re- e Provide DCPS with new ways to identify students 
DCPS serves 54,000 students enrolled at risk of not re-enrolling. 

e 49,103 enrolled in DCPS e Provide DCPS with information on which SEL 
competencies or aspects of school climate to 
focus on for improving individual students’ re- 
enrollment. 

DCPS is District of Columbia Public Schools. ELA is English language arts. PARCC is Partnership for Assessment of Readiness for College and Career. SEL is 


social and emotional learning. 
® Based on DCPS’s one-year capital commitment update (DCPS, 2018). 


This study provides a range of information that DCPS can use to better achieve its six strategic goals, with a 
particular focus on Goal 4, which directly relates to SEL competencies and school climate. In particular, the study 
provides key information on the types of students who are not feeling loved, challenged, and prepared as well as 
the types of schools they attend. Such findings may suggest how DCPS can best target its investments to make the 
biggest difference to progress toward Goal 4, which is especially pertinent because investment in SEL across the 
district is on the rise. In addition to supporting a district-level SEL team, the district has hired school-based SEL 
leads, who work with principals to create school-level SEL and climate goals and, in some cases, have begun to 
administer additional short surveys around these topics at the beginning, middle, and end of the school year. 
Because these grassroots efforts are not yet standardized across schools, a better understanding of its Panorama 
SEL data may allow the district to create well-informed, universal SEL and climate practices or programming to 
attain its goals. 


While Goals 1, 2, 3,5, and 6 are longer-term and less explicitly tied to SEL and school climate, the study will provide 
information about how boosting SEL competencies and improving school climate could be an avenue for achieving 
them. 
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Appendix B. Methods 


This appendix describes the data, samples, weights, and analysis methods for the Regional Educational Laboratory 
Mid-Atlantic study to explore the properties and uses of social and emotional learning (SEL) survey data to inform 
evaluation and track progress toward District of Columbia Public School (DCPS) goals. 


Data 


The key data sources for this study are DCPS’s Panorama Education (Panorama) surveys and administrative 
records. 


Panorama surveys. The Panorama survey is administered annually to students, teachers, and parents. All students 
between grades 3 and 12 who attended at least one day of school are eligible to complete the survey, while all 
parents and teachers are eligible. If a parent has multiple children in a single school, the parent is asked to 
complete one survey with the oldest child in mind (DCPS, 2020). If the parent has multiple children enrolled in 
different schools, the parent is asked to complete one survey for each school. The student survey assesses four 
SEL competencies—perseverance, self-management, self-efficacy, and social awareness—and three aspects of 
school climate—rigorous expectations, sense of belonging, and student satisfaction. The number of items in each 
topic ranges from four to eight, and the version for grades 3—5 includes fewer items for perseverance and sense 
of belonging (table B1). DCPS also surveys teachers and parents to assess their perceptions of students’ SEL 
competencies and personal experiences of school climate that are related to perseverance and rigorous 
expectations. The teacher surveys include questions about these selected competencies of their students as a 
group, as well as their own engagement and professional learning about SEL. The parent surveys include questions 
about their child’s SEL competencies, as well as questions about their own experience with the school and 
individual teachers. When referring to individual respondents’ reports, this study uses the term school 
experiences. Averaging the measures of school experiences across respondents at a school provides a measure of 
a school’s climate. 


Table B1. Number of items in DCPS’s customized Panorama Education survey 


Self- Self- Social Rigorous Sense of Student 
Respondent type Perseverance management efficacy awareness expectations belonging satisfaction 
Student 
Grades 3-5 4 8 5 4 8 
Grades 6-12 5 8 5 5 8 
Parent 6 na na na 3 na na 
Teacher 5 na na na 5 na na 


DCPS is District of Columbia Public Schools. na is not applicable. 


For each SEL competency and school climate measure, the associated scales are calculated by averaging across 
numerical values that correspond to each possible response category. Each response category is assigned a value 
between 1 and 5, so the resulting scales range from 1 to 5 points.’” Following DCPS’s methodology, this study 
counted the scale as non-missing if a respondent completed at least two items in the scale. 


The administration of the Panorama survey was directly impacted by the outbreak of the COVID-19 pandemic. 
The survey, which is typically administered online to students in school in early March, was only available online 
for students to complete at home once students could no longer attend school in person, resulting in a lower 
response rate for students (51 percent, which is down nearly 20 percent from the previous year). For all 


1’ The student satisfaction items have four response categories. When reporting findings, the study team rescaled the student satisfaction 
scale to range from 1 to 5, so it was comparable to the other scales. To do so, the study team rescaled the values linearly so that a 2 on the 
original scale mapped to a 2.33 on the updated scale, a 3 mapped to a 3.66, and a 4 mapped to a 5. 
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respondents, the survey was likely not a top priority given the pandemic’s upheaval to normal life, and those who 
did respond might have responded differently as a result. For these reasons, our main analyses did not include 
the 2019/20 survey data. 


Over the two school years included in the study (2017/18—2018/19), there were 39,791 unique students enrolled 
and 30,462 who responded to the Panorama survey (table B2). Analyses that include parent and teacher samples 
were also run over only two years of data (2017/18—2018/19). Over this period, there were 4,273 unique teachers 
who responded and 12,216 responses from parents on the measures used in this study. 


Table B2. Sample sizes by respondent type and school year 


Sample description RYol see) YZ =F-18 ()) Sample size 
Unique students who were eligible for the survey 2017/18-2018/19 39,791 
2017/18 31,452 
2018/19 31,926 
Unique student respondents for the survey 2017/18-2018/19 30,462 
2017/18 21,385 
2018/19 22,208 
Unique teacher respondents for the survey 2017/18-2018/19 4,273 
measures used in this study 2017/18 3,109 
2018/19 3,130 
Parent responses to the survey measures used in 2017/18-2018/19 12,216 
this study? 2017/18 5,734 
2018/19 6,482 


® These calculations do not represent unique parent respondents, because they could include multiple responses from the same person. 
Source: Authors’ analyses based on survey and administrative data provided by the District of Columbia Public Schools, 2017/18 to 2018/19. 


Administrative data. The student administrative data files included information for all DCPS students from fall 
2017 through spring 2020. The data included demographic information, such as gender, special education status, 
English learner status, grade level, race/ethnicity, and whether they are at risk (per DCPS’s classification; table B3). 
They also included academic measures, such as absences and in-seat attendance rates, suspensions, grade 
progression, enrollment status, math and English language arts (ELA) summative assessment proficiencies on the 
Partnership for Assessment of Readiness for College and Careers (PARCC) or Multi-State Alternate Assessment 
(MSAA), predicted college readiness based on SAT or Preliminary SAT (PSAT) scores, and credits earned.7® In 
addition, the data files included teacher demographic information (such as gender, race/ethnicity, and 
employment duration) and school characteristics (such as DC School Transparency and Reporting [STAR] rating 
and geographic ward). 


Similar to the Panorama survey data, the outbreak of the COVID-19 pandemic affected the availability of 
administrative data during the 2019/20 school year. Certain annual standardized tests were canceled in spring 
2020, including the PARCC exams and some spring administrations of the SAT and PSAT. 


18 MISAA scores were only received in school year 2018/19. 
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Table B3. Student characteristics and outcomes used in the study 


Characteristic Description 


Demographic variables 


Female Whether a student was female 

Age Age of a student as of September 30th for that academic year 

Grade level Which grade (3-12) a student was in at the time of enrollment for that academic year 
Survey year Which academic year data was from (2017/18 or 2018/19) 


Race/ethnicity Whether a student was 
e Black (non-Hispanic) 
e Hispanic 
e White (non-Hispanic) 
e Other 

At-risk status Whether a student was flagged as at risk. “In the District of Columbia, at-risk is defined as a student who 
possesses one of the following characteristics at any point during the given school year: eligibility for 
Temporary Assistance for Needy Families (TANF), eligibility for Supplemental Nutrition Assistance Program 
(SNAP), identification as homeless by the student’s school or other community partners, under the care of the 
Child and Family Services Agency (CFSA, also known as foster care), and/or overage (high school only). A high 
school student is overage if he or she is at least one year older than the expected age for their grade” (District 
of Columbia Office of the State Superintendent of Education, 2020). 

Special education Whether a student received special education services 


status 

English learner Whether a student was an English learner 

status 

Ward In which of eight geographic areas, called a ward, a student attended school 

STAR rating To which of five rating scores a student’s school was assigned. “The School Transparency and Reporting 


(STAR) Framework is the accountability framework for public schools in the District of Columbia. The STAR 
Framework uses common measures of performance across schools and is comprised of multiple data points 
from multiple data sources. Schools receive a summative STAR Rating ranging from 1 to 5 stars, with 5 being 
the highest” (District of Columbia Office of the State Superintendent of Education, 2020). 

Academic behavior measures 

In-seat attendance The number of days a student attended school during the school year divided by the number of days they 
were enrolled (at the school where students spent the majority of time during the school year) 

Chronically absent Whether a student was chronically absent (absent for 10 percent or more of the school year) 


Number of The number of times a student was suspended during the school year 

suspensions 

Suspended Whether a student was ever suspended during the school year 

Progressed Whether a student who was enrolled in 2017/18 progressed as expected in 2018/19 and 2019/20. For 
successfully example, if the student was in grade 3 in 2017/18, whether they were in grade 4 in 2018/19 and grade 5 in 


2019/20. For students in grade 12, this would be whether they graduated. Progression also included students 
who skipped a grade (for example, a student in grade 3 in 2017/18 and grade 5 in 2018/19). 
Number of credits The number of credits high school students were behind for a given grade level 


behind 

Successful re- Whether a student who was enrolled in 2017/18 remained enrolled through later school years 

enrollment 

Red status for Whether a student was flagged as being 4.25+ credits behind in order to graduate on time, assuming students 


grade progression earn 6 credits per year in high school 
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Table B3. Student characteristics and outcomes used in the study (continued) 


Characteristic DY =KXo del d(oya) 


Academic proficiency measures 


Number of AP credits |The number of AP credits a student completed/earned during the school year 
earned 


Earned AP credit Whether a student completed/earned any AP credits during the school year 
Proficient/college ready In 2017/18, whether a grade 3-10 student was proficient in math/ELA based on their PARCC score (a 
in math and ELA score of 4+ ona scale of 1 to 5) and whether a grade 11-12 student was college ready in 


math/reading/writing based on their SAT score (a score of 530+ in math and 480+ in EBRW). In 2018/19, 
whether a grade 3-10 student was proficient in math/ELA based on their PARCC score or whether a 
grade 3-8 student was proficient in math/ELA based on their MSAA score (a score of 3+ on a scale of 1to 
4) and whether a grade 11-12 student was college ready in math/reading/writing based on their SAT 
score. Students in applicable grades in these years took at most either PARCC or MSAA, not both. 

Proficient on the PARCC Whether a student was proficient in math based on their PARCC score (a score of 4+ on a scale of 1 to 5) 

in math 

Proficient on the PARCC Whether a student was proficient in ELA based on their PARCC score (a score of 4+ on a scale of 1 to 5) 

in ELA 

College ready inmath Whether a student was college ready in math based on their best math SAT score (530+) or their best 
math PSAT score (PSAT college-ready score threshold is based on grade level) 

College ready in ELA Whether a student was college ready in reading/writing based on their best EBRW SAT score (480+) or 
their best EBRW PSAT score (PSAT college-ready score threshold is based on grade level) 

Math and ELA The average math and ELA achievement tests score of the student 

achievement 

Outcomes based on SEL competencies and school experiences 


Student Loved, Whether or not a student felt loved, challenged, and prepared, based on the student’s reports of SEL 
Challenged, and competencies and school experiences. In the index, “loved” is captured by items on sense of belonging, 
Prepared Index “challenged” is captured by items on rigorous expectations, and “prepared” is captured by items on 


perseverance, self-management, and self-efficacy. The index is formed in two steps. First, for each 
component, students are assigned a value of 1 if their average score on the scales associated with that 
component exceeds 3.5 on 1- to 5-point scale, and they are assigned a value of 0 otherwise. For 
example, if a student’s score on the sense of belonging scale exceeded 3.5, they would be assigned a 
value of 1 on the loved component, indicating that they feel loved. Second, if they are assigned a value 
of 1 for all three components, then they are assigned a value of 1 for the Student Loved, Challenged, and 
Prepared Index. If they are assigned a value of 0 for any component, they are assigned a value of 0 for 
the Index. This student outcome variable was developed to measure progress toward DCPS’s strategic 
goal of having 100 percent of students feel loved, challenged, and prepared by 2022. 

AP is Advanced Placement. DCPS is District of Columbia Public Schools. EBRW is Evidence-Based Reading and Writing. ELA is and English language arts. MSAA 


is Multi-State Alternate Assessment. PARCC is Partnership for Assessment of Readiness for College and Careers. PSAT is Preliminary SAT. SEL is social and 
emotional learning. STAR is School Transparency and Reporting. 


Data acquisition 

DCPS generated a unique district identification number (StudentID) for all enrolled students and provided the 
study team with the data associated with each student for 2017/18-2019/20, identified by StudentID. In addition, 
teacher and parent SEL files had unique respondent identification numbers. The data did not include student, 
teacher, or parent names, addresses, or social security numbers, but the study team took steps to protect the 
data that included StudentIDs. 


Sample 


The sample sizes differed by analysis (table B4). In some cases, the sample sizes counted the same students once 
in each observed year over multiple years, such as the analyses of students’ SEL competencies and school 
experiences for research question 1. Therefore, students in these analyses could be counted multiple times if they 
appeared in multiple years. In other cases, the analyses required that individual students have data in multiple 
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years, such as the analyses of the change in SEL competencies and school experiences across years for research 
question 2. 


Table B4. Sample sizes by research question and analysis sample for the main analyses 


Research question (RQ) Analysis sample Sample size 
RQ1. Differences in students’ SEL Students who were eligible for the Panorama 43,578 student-year observations 
competencies and school experiences across survey and completed it in 2017/18 or 6,222 teacher-year observations 
grades 2018/19 
RQ2. Differences in year-to-year changes in Students who were eligible for the Panorama 13,131 students 
measures across schools survey and completed it in 2017/18 and 
2018/19 
RQ3. Correlations between predictors and Students who were eligible for the Panorama 18,152 students 
outcomes and classification accuracy of survey and completed it in 2017/18 and had 
predictors non-missing data on each predictor 
RQ3. Correlations between early literacy Students who were in grade 3 in 2017/18 and 6,665 student-year observations 
and ELA achievement completed the Panorama survey in 2017/18 or 


were in grade 3 in 2018/19 and completed the 

Panorama survey in 2018/19 
RQ3. Correlations between grade 8 Students who were in grade 8 in 2017/18, 1,394 students 
measures and grade 9 measures eligible for the Panorama survey and 

completed it in 2017/18, and had non-missing 

data on each predictor 


RQ4. Correlations between school-level Students, parents, and teachers who were 43,116 student-year observations, 
averages of student, parent, and teacher eligible for the Panorama survey and 12,216 parent-year observations, 
reports completed at least one of the measures used 6,222 teacher-year observations, 


in this study in 2017/18 or 2018/19; schools 233 schools 
that had at least one student complete the 
Panorama survey in 2017/18 or 2018/19 
RQ4. Differences between student, parent, Students, parents, and teachers who were 43,116 student-year observations, 
and teacher reports eligible for the Panorama survey and 12,216 parent-year observations, 
completed at least one of the measures used 6,222 teacher-year observations, 
in this study in 2017/18 or 2018/19; schools 233 schools 
that had at least one student complete the 
Panorama survey in 2017/18 or 2018/19 


ELA is English language arts. SEL is social and emotional learning. 
Source: Authors’ analyses based on survey and administrative data provided by the District of Columbia Public Schools, 2017/18 to 2018/19. 


Analysis methods 


Accounting for survey nonresponse. 


The study team explored and accounted for nonresponse bias for the SEL competencies and school experiences 
in the student and teacher surveys. The usefulness of a survey hinges on obtaining responses from a 
representative set of respondents; if only a select type of person responds, then the resulting findings could paint 
a misleading picture of the population of interest. The study team explored response rates and the potential for 
nonresponse bias, and it constructed nonresponse weights to mitigate that bias. 


Response rates. The study team calculated the following rates for each analysis sample and survey measure 
(tables BS and B6): 


e Unit response rate, which is the number of respondents who took the survey out of those who were eligible to 
take the survey. 
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e Item response rate, which is the number of respondents who had non-missing values for a scale out of those 
who took the survey. 


e Overall response rate, which is the number of respondents who had non-missing values for a scale out of those 
who were eligible to take the survey.?? 


For students, the response rates were calculated for six different samples, which included three samples based 
on students who were eligible to take the survey in individual years (for example, 2017/18) and three samples 
based on whether students were eligible to take the survey in two years (for example, both 2017/18 and 2018/19). 
For the samples based on individual years, the overall response rates ranged from 50.7 percent to 69.1 percent. 
For samples based on two years, the overall response rates ranged from 37.3 percent to 55.0 percent. The results 
revealed that unit nonresponse—rather than item nonresponse—drove the overall response rates for each scale. 


For teachers, the response rates were calculated for two different samples: teachers who were eligible to take 
the survey in 2017/18 and teachers who were eligible to take it in 2018/19. The overall response rates ranged 
from 59.7 to 77.1. As with the student survey, unit nonresponse drove the overall response rates for each scale. 


Nonresponse bias analysis. Because the overall response rates were less than 85 percent for each of our samples, 
a conventional standard for nonresponse bias to be considered negligible, the study team conducted a 
nonresponse bias analysis. The team compared the characteristics of the students and teachers who responded 
to the survey to the full sample of those who were eligible. For students, the study has administrative records on 
a number of other characteristics, including their grade level, gender, English learner status, special education 
status, and in-seat attendance, as well as the number of suspensions they received and whether they are at risk 
(as defined by the DCPS indicator). For teachers, the analyses included information on their gender, race/ethnicity, 
and level of experience, as well as information on their school—including the type of school, its STAR rating, the 
geographical ward, the racial/ethnic composition of its students, the average math and ELA achievement tests 
scores of the students, the average number of suspensions per student, and the average in-seat attendance for 
students. These variables were all missing in less than 5 percent of the sample cases. 


A nonresponse bias analysis requires having at least one characteristic that is strongly related to each survey 
measure. To assess the relationship between the survey measures and the characteristics, the study team 
calculated the correlations between them. Because all the individual characteristics had relatively low correlations 
with the survey measures, the study team constructed a composite variable of characteristics that would be more 
highly correlated with the survey measures. Compared to the individual measures, the composite variables were 
more correlated with the survey measures, because the composite variables summarized information from 
multiple individual measures, which can reduce measurement error. Specifically, to form the composite variable, 
the study team conducted an ordinary least squares regression of each of the survey measures on the group of 
student and teacher characteristics. The study team then formed the composite variable as the predicted value 
of the survey measure for each student in the sample. For the samples based on individual years of data, the 
correlations between the student survey measures and the composite variable ranged from 0.22 to 0.39 (tables 
B7-B13). The study team deemed that these correlations were strong because they approached the rule-of-thumb 
cutoff for a strong correlation of 0.25. For the samples based on two years of data, the correlations between 
changes in the student survey measures and the composite variable ranged from 0.07 to 0.28 (tables B14—B20). 
Although these correlations were not as strong, the study team is not aware of evidence on variables that have a 
stronger correlation with changes in measures of SEL competencies and school experiences. The correlations 
between the teacher survey measures and the composite variable ranged from 0.23 to 0.40 (tables B21—B22). 


19 To be considered not missing, a respondent needed to answer at least two items for the scale. 
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Because they approached the 0.25 cutoff for a strong correlation, the study team deemed that these correlations 
were strong. 


For the composite variable and each of these characteristics, the study team calculated the difference in standard 
deviation units between respondents to the survey and the full samples of students and teachers who were 
eligible (tables B7—B22). At least one difference exceeded 0.05 standard deviations for each survey measure, 
providing evidence of nonresponse bias. Notably, the tables reveal that, because students in higher grades 
responded less frequently, they were underrepresented in the survey sample—compared to those in lower 
grades—for each survey measure. 


Nonresponse response weights. Given the evidence of nonresponse bias, the study team constructed 
nonresponse weights for students and teachers. Nonresponse weights help to ensure that the results are 
statistically representative, reflecting the composition of students in DCPS by giving higher weight to the types of 
respondents who are less likely to respond. To form nonresponse weights, the team adopted the following steps: 


1. Using a probit model, the study team estimated the probability of completing the survey as a function of the 
student characteristics listed in tables B7—B22. 

2. Using estimates from each of the probit models, the study team calculated the propensity of being in the 
index sample for each respondent in the sample (Propensity_i). Next, the study team calculated the deciles 
of the distribution of propensities and assigned each respondent the average value within their decile 
(Propensity_decile_i). 

3. Finally, the study team formed nonresponse weights by taking the inverse of the average decile value for 
each respondent (1/Propensity_decile_i). 


To assess the effectiveness of the weights, the study team compared the difference in the composite of covariates 
between the survey sample and the original sample with and without nonresponse weights (table B23). For both 
the student and teacher surveys, the absolute difference with weights was less than 0.05 standard deviations for 
the individual years on which the study focuses (2017/18 or 2018/19)—a rule of thumb for an acceptable 
difference. For the analyses that the study focused on (that involved students who were eligible in two years: 
2017/18 and 2018/19), all but one absolute difference was 0.07 standard deviations or less. The remaining 
difference was 0.09 standard deviations. For the samples the study did not use in the main analyses, the 
differences tended to be greater. 
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Table B5. Response rates (student survey) 


Type of data the study 2017/18 and 2018/19 and 2017/18 and 
attempted to collect 2017/18 2018/19 2019/20 PAIN EY As) 2019/20 2019/20 
Perseverance 

Unit response rate 68.0 69.6 51.3 55.7 41.8 39.4 
Item response rate 95.7 98.5 99.1 94.3 97.6 94.6 
Overall response rate 65.1 68.5 50.9 52.5 40.8 37.3 
Self-management 

Unit response rate 68.0 69.6 51.3 55.7 41.8 39.4 
Item response rate 96.4 97.8 98.7 94.6 96.9 95.6 
Overall response rate 65.6 68.0 50.7 52.6 40.5 37.6 
Self-efficacy 

Unit response rate 68.0 69.6 51.3 55.7 41.8 39.4 
Item response rate 96.6 97.8 99.0 94.8 97.0 95.9 
Overall response rate 65.7 68.0 50.8 52.8 40.5 37.8 
Social awareness 

Unit response rate 68.0 69.6 51.3 55.7 41.8 39.4 
Item response rate 96.5 98.0 98.9 94.7 97.2 95.5 
Overall response rate 65.6 68.2 50.7 52.7 40.6 37.6 
Rigorous expectations 

Unit response rate 68.0 69.6 51.3 55.7 41.8 39.4 
Item response rate 97.5 99.2 99.5 96.8 98.7 97.3 
Overall response rate 66.3 69.0 51.1 53.9 41.2 38.3 
Student satisfaction 

Unit response rate 68.0 69.6 51.3 55.7 41.8 39.4 
Item response rate 99.4 99.4 99.8 98.9 99.3 99.3 
Overall response rate 67.6 69.1 51.2 55.0 41.5 39.1 
Sense of belonging 

Unit response rate 68.0 69.6 51.3 55.7 41.8 39.4 
Item response rate 98.9 99.3 99.5 98.2 98.9 98.3 
Overall response rate 67.3 69.1 51.1 54.7 41.3 38.7 


Note: The table shows the unit, item, and overall response rates to the student survey for each of the seven survey scales. The item response rate is the 
percentage of observations for which each scale was missing among those who responded to the survey. 
Source: Authors’ analyses based on survey and administrative data provided by the District of Columbia Public Schools, 2017/18 to 2019/20. 


Table B6. Response rates (teacher survey) 


Type of data the study attempted to collect 2017/18 2018/19 
Perseverance 

Unit response rate 60.1 77.3 
Item response rate 99.3 99.7 
Overall response rate 59.7 77.1 
Rigorous expectations 

Unit response rate 60.1 77.3 
Item response rate 99.3 99.8 
Overall response rate 59.7 77.1 


Note: The table shows the unit, item, and overall response rates to the teacher survey for each of the two survey scales used in the study. The item response 
rate is the percentage of observations for which each scale was missing among those who responded to the survey. 
Source: Authors’ analyses based on survey and administrative data provided by the District of Columbia Public Schools, 2017/18 to 2018/19. 
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Table B7. Exploration of nonresponse bias for perseverance (student survey) 
Mean for original study 


sample (with standard Difference in standard Correlation with 
Mean for survey sample deviation) deviation units perseverance scale 
Covariates and 
units 2017/18 2018/19 2019/20 2017/18 2018/19 2019/20 2017/18 2018/19 2019/20 2017/18 2018/19 2019/20 
Female 0.50 0.50 0.51 0.49 0.49 0.49 0.03 0.03 0.03 0.05 0.02 0.00 
(0.50) (0.50) (0.50) 
Grade level 
3 0.16 0.15 0.17 0.13 0.13 0.13 0.09 0.07 0.13 0.10 0.10 0.10 
(0.34) (0.34) (0.33) 
4 0.16 0.15 0.17 0.13 0.13 0.12 0.09 0.08 0.13 0.08 0.09 0.07 
(0.34) (0.33) (0.33) 
5 0.15 0.15 0.16 0.12 0.12 0.11 0.09 0.08 0.13 0.05 0.05 0.06 
(0.32) (0.32) (0.32) 
6 0.08 0.10 0.11 0.08 0.09 0.09 0.00 0.02 0.06 0.00 -0.02 -0.04 
(0.27) (0.29) (0.29) 
7 0.09 0.09 0.11 0.08 0.08 0.09 0.03 0.04 0.07 -0.05 -0.07 -0.08 
(0.27) (0.27) (0.29) 
8 0.07 0.09 0.09 0.07 0.08 0.08 0.00 0.03 0.04 = -0.05 -0.08 -0.08 
(0.26) (0.27) (0.28) 
9 0.08 0.07 0.06 0.10 0.10 0.12 -0.09 -0.10 -0.17 -0.08 -0.05 -0.06 
(0.31) (0.31) (0.32) 
10 0.07 0.06 0.04 0.10 0.09 0.09 -0.09 -0.10 -0.17 -0.07 -0.06 -0.04 
(0.30) (0.29) (0.28) 
11 0.07 0.07 0.04 0.09 0.08 0.08  -0.06 -0.07 -0.13 -0.05 -0.04 -0.02 
(0.28) (0.28) (0.26) 
12 0.07 0.06 0.04 0.09 0.09 0.08 -0.10 -0.09 -0.16 -0.01 -0.01 -0.01 


(0.29) (0.28) (0.27) 
English learner 0.12 0.12 0.13 0.12 0.12 0.14 0.01 -0.02 -0.03 0.01 0.03 0.02 


student (0.32) (0.33) (0.34) 

Special 014 014 014 0.16 0.17 0.17 -0.06 -007 -0.07 -0.03 -0.02 -0.02 
education (0.37) (0.37) (0.37) 

student 

Fraction ofdays 0.09 0.09 0.07 O15 O15 013 -0.27 0.27 -0.32 -0.08 0.06 -0.05 
absent (0.20) (0.21) (0.19) 

Numberof 014 0.16 0.07 018 0.20 0.09 -0.05 -0.05 -0.07 0.06 0.05 -0.03 
suspensions (0.65) (0.74) (0.43) 


At-risk status 0.42 O43 042 042 #4048 O48 O00 -010 -0.13 0.008 0.08 0.09 
(0.49) (0.50) (0.50) 

Composite of 3.78 3.78 3.77 3.75 3.75 3.73 0.19 O13 £40.22 «0.24 «0.25 ~~» 0.24 

covariates (0.19) (0.19) (0.18) 


Note: This table explores nonresponse bias through the mean, standard deviation, and correlation with the scale across covariates. 
Source: Authors’ analyses based on survey and administrative data provided by the District of Columbia Public Schools, 2017/18 to 2019/20. 
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Table B8. Exploration of nonresponse bias for self-management scale (student survey) 
Mean for original study 


sample (with standard Difference in standard Correlation with self- 
Mean for survey sample deviation) deviation units management scale 
Covariates and 
units 2017/18 2018/19 2019/20 2017/18 2018/19 2019/20 2017/18 2018/19 2019/20 2017/18 2018/19 2019/20 
Female 0.50 0.50 0.51 0.49 0.49 0.49 0.03 0.03 0.04 0.10 0.08 0.06 
(0.50) (0.50) (0.50) 
Grade level 
3 0.16 0.16 0.17 0.13 0.13 0.13 0.10 0.07 0.14 0.02 0.04 0.03 
(0.34) (0.34) (0.33) 
4 0.16 0.16 0.17 0.13 0.13 0.12 0.09 0.09 0.13 0.03 0.04 0.04 
(0.34) (0.33) (0.33) 
5 0.15 0.15 0.16 0.12 0.12 0.11 0.10 0.09 0.13 0.00 0.00 0.02 
(0.32) (0.32) (0.32) 
6 0.08 0.10 0.11 0.08 0.09 0.09 0.00 0.02 0.06 0.00 0.00 -0.02 
(0.27) (0.29) (0.29) 
7 0.09 0.09 0.11 0.08 0.08 0.09 0.02 0.03 0.07 -0.02 -0.06 -0.05 
(0.27) (0.27) (0.29) 
8 0.07 0.09 0.09 0.07 0.08 0.08 0.00 0.02 0.04 ~=-0.03 -0.03 -0.03 
(0.26) (0.27) (0.28) 
9 0.08 0.07 0.06 0.10 0.10 0.12 -0.09 -0.10 -0.17 -0.03 -0.02 -0.02 
(0.31) (0.31) (0.32) 
10 0.07 0.06 0.04 0.10 0.09 0.09 -0.09 -0.10 -0.17 -0.03 -0.02 0.00 
(0.30) (0.29) (0.28) 
11 0.07 0.06 0.04 0.09 0.08 0.08  -0.06 -0.07 -0.13 0.01 0.00 0.01 
(0.28) (0.28) (0.26) 
12 0.07 0.06 0.04 0.09 0.09 0.08  -0.10 -0.09 -0.16 0.04 0.02 0.00 


(0.29) (0.28) (0.27) 
English learner 0.12 0.12 0.13 0.12 0.12 0.14 0.01 -0.02 -0.03 -0.02 -0.03 -0.04 


student (0.32) (0.33) (0.34) 

Special §§ 014 0.14 0.14 016 017 0.17 -0.06 -0.07 -0.07 010 011 -0.10 
education (0.37) (0.37) (0.37) 

student 

Fractionof 0.09 0.09 0.07 O15 O15 013 -0.27 -0.27 -032 O11 012 -0.09 
days absent (0.20) (0.21) (0.19) 

Numberof 014 0.16 0.06 018 0.20 009 -0.06 0.06 -0.07 -014 014 -0.10 
suspensions (0.65) (0.74) (0.43) 


At-risk status 0.42 0.43 O42 O42 048 +4048 O00 -0.10 -013 -009 -0.10 -0.11 
(0.49) (0.50) (0.50) 

Composite of 3.77 3.80 3.78 3.74 3.76 3.74 140.18 019 £40.23 0.24 0.24 ~» 0.22 

covariates (0.20) (0.20) (0.18) 


Note: This table explores nonresponse bias through the mean, standard deviation, and correlation with the scale across covariates. 
Source: Authors’ analyses based on survey and administrative data provided by the District of Columbia Public Schools, 2017/18 to 2019/20. 
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Table B9. Exploration of nonresponse bias for self-efficacy (student survey) 
Mean for original study 


sample (with standard Difference in standard Correlation with self- 
Mean for survey sample deviation) deviation units efficacy scale 
Covariates and 
units 2017/18 2018/19 2019/20 2017/18 2018/19 2019/20 2017/18 2018/19 2019/20 2017/18 2018/19 2019/20 
Female 0.50 0.50 0.51 0.49 0.49 0.49 0.03 0.03 0.04 -0.02 -0.04 -0.06 
(0.50) (0.50) (0.50) 
Grade level 
3 0.16 0.15 0.17 0.13 0.13 0.13 0.09 0.07 0.14 0.11 0.12 0.10 
(0.34) (0.34) (0.33) 
4 0.16 0.16 0.17 0.13 0.13 0.12 0.09 0.08 0.13 0.12 0.12 0.10 
(0.34) (0.33) (0.33) 
5 0.15 0.15 0.16 0.12 0.12 0.11 0.09 0.09 0.14 0.10 0.08 0.10 
(0.32) (0.32) (0.32) 
6 0.08 0.10 0.11 0.08 0.09 0.09 0.00 0.02 0.06 0.01 0.01 -0.02 
(0.27) (0.29) (0.29) 
7 0.09 0.09 0.11 0.08 0.08 0.09 0.03 0.03 0.07 -0.05 -0.07 -0.07 
(0.27) (0.27) (0.29) 
8 0.07 0.09 0.09 0.07 0.08 0.08 0.00 0.02 0.04 -0.06 -0.08 -0.07 
(0.26) (0.27) (0.28) 
9 0.08 0.07 0.06 0.10 0.10 0.12 -0.09 -0.10 -0.17 -0.10 -0.09 -0.10 
(0.31) (0.31) (0.32) 
10 0.07 0.06 0.04 0.10 0.09 0.09 -0.09 -0.10 -0.17 -0.11 -0.09 -0.07 
(0.30) (0.29) (0.28) 
11 0.07 0.07 0.04 0.09 0.08 0.08  -0.06 -0.07 -0.13 -0.09 -0.09 -0.06 
(0.28) (0.28) (0.26) 
12 0.07 0.06 0.04 0.09 0.09 0.08 -0.10 -0.09 -0.16 -0.05 -0.04 -0.05 


(0.29) (0.28) (0.27) 
English learner 0.12 0.12 0.13 0.12 0.12 0.14 0.01 -0.02 -0.03 -0.01 -0.01 -0.02 


student (0.32) (0.33) (0.34) 

Special = 0.14 0.14 0.14 016 017 017 -0.06 -0.07 -0.07 005 0.05 -0.04 
education (0.37) (0.37) (0.37) 

student 

Fraction ofdays 0.09 0.09 0.07 O15 O15 013 -0.27 0.27 -032 -012 010 -0.10 
absent (0.20) (0.21) (0.19) 

Numberof 014 016 0.06 018 0.20 0.09 -0.05 0.05 -0.07 -0.05 -0.05 -0.03 
suspensions (0.65) (0.74) (0.43) 


At-risk status 0.42 O43 #042 #42042 #42048 O48 +(O.00 -010 -013 002 £0.01 #0.02 
(0.49) (0.50) (0.50) 

Composite of 3.70 3.70 3.69 364 366 3.61 £0.24 40.20 0.35 £0.27. «0.27 ~~ 0.26 

covariates (0.24) (0.23) (0.23) 


Note: This table explores nonresponse bias through the mean, standard deviation, and correlation with the scale across covariates. 
Source: Authors’ analyses based on survey and administrative data provided by the District of Columbia Public Schools, 2017/18 to 2019/20. 
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Table B10. Exploration of nonresponse bias for social awareness (student survey) 
Mean for original study 


sample (with standard Difference in standard Correlation with social 
Mean for survey sample deviation) deviation units awareness scale 
Covariates and 
units 2017/18 2018/19 2019/20 2017/18 2018/19 2019/20 2017/18 2018/19 2019/20 2017/18 2018/19 2019/20 
Female 0.50 0.50 0.51 0.49 0.49 0.49 0.03 0.03 0.04 0.10 0.07 0.06 
(0.50) (0.50) (0.50) 
Grade level 
3 0.16 0.15 0.17 0.13 0.13 0.13 0.10 0.07 0.14 0.11 0.14 0.11 
(0.34) (0.34) (0.33) 
4 0.16 0.16 0.17 0.13 0.13 0.12 0.09 0.08 0.13 0.09 0.11 0.09 
(0.34) (0.33) (0.33) 
5 0.15 0.15 0.16 0.12 0.12 0.11 0.09 0.09 0.13 0.05 0.05 0.08 
(0.32) (0.32) (0.32) 
6 0.08 0.10 0.11 0.08 0.09 0.09 0.00 0.02 0.06 -0.02 -0.02 -0.06 
(0.27) (0.29) (0.29) 
7 0.09 0.09 0.11 0.08 0.08 0.09 0.03 0.03 0.07 -0.08 -0.09 -0.08 
(0.27) (0.27) (0.29) 
8 0.07 0.09 0.09 0.07 0.08 0.08 0.00 0.02 0.04 -0.06 -0.09 -0.08 
(0.26) (0.27) (0.28) 
9 0.08 0.07 0.06 0.10 0.10 0.12 -0.09 -0.10 -0.17 -0.08 -0.08 -0.08 
(0.31) (0.31) (0.32) 
10 0.07 0.06 0.04 0.10 0.09 0.09 -0.09 -0.10 -0.17 -0.08 -0.06 -0.04 
(0.30) (0.29) (0.28) 
11 0.07 0.06 0.04 0.09 0.08 0.08  -0.06 -0.07 -0.13 -0.04 -0.03 -0.02 
(0.28) (0.28) (0.26) 
12 0.07 0.06 0.04 0.09 0.09 0.08  -0.10 -0.08 -0.16 0.00 -0.01 -0.02 


(0.29) (0.28) (0.27) 
English learner 0.12 0.12 0.13 0.12 0.12 0.14 0.01 -0.02 -0.03 0.03 0.00 0.01 


student (0.32) (0.33) (0.34) 

Special § 0.14 0.14 0.14 016 017 0.17 -0.06 -0.07 -0.07  -0.06 0.06 -0.07 
education (0.37) (0.37) (0.37) 

student 

Fractionofdays 0.09 0.09 0.07 O15 O15 013 -0.27 -0.27 -032 014 015 -013 
absent (0.20) (0.21) (0.19) 

Numberof 014 016 0.06 018 0.20 009 -0.05 0.05 -0.07 -013 -012 -0.09 
suspensions (0.65) (0.74) (0.43) 


At-risk status 0.42 O43 042 #042 #429048 O48 O00 -010 -0.13 -0.04 -0.05 -0.06 
(0.49) (0.50) (0.50) 

Composite of 3.81 3.80 3.79 3.76 3.75 3.73 0.23 0.23 0.32 0.27. 0.28 0.25 

covariates (0.22) (0.22) (0.21) 


Note: This table explores nonresponse bias through the mean, standard deviation, and correlation with the scale across covariates. 
Source: Authors’ analyses based on survey and administrative data provided by the District of Columbia Public Schools, 2017/18 to 2019/20. 
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Table B11. Exploration of nonresponse bias for rigorous expectations (student survey) 
Mean for original study 


sample (with standard Difference instandard Correlation with rigorous 
Mean for survey sample deviation) deviation units expectations scale 
Covariates and 
units 2017/18 2018/19 2019/20 2017/18 2018/19 2019/20 2017/18 2018/19 2019/20 2017/18 2018/19 2019/20 
Female 0.50 0.50 0.51 0.49 0.49 0.49 0.02 0.03 0.03 0.06 0.04 0.03 
(0.50) (0.50) (0.50) 
Grade level 
3 0.16 0.15 0.17 0.13 0.13 0.13 0.10 0.07 0.13 0.17 0.15 0.12 
(0.34) (0.34) (0.33) 
4 0.16 0.15 0.17 0.13 0.13 0.12 0.09 0.08 0.13 0.16 0.17 0.15 
(0.34) (0.33) (0.33) 
5 0.15 0.15 0.16 0.12 0.12 0.11 0.09 0.08 0.13 0.15 0.15 0.16 
(0.32) (0.32) (0.32) 
6 0.08 0.10 0.11 0.08 0.09 0.09 0.00 0.02 0.06 -0.01 -0.02 -0.05 
(0.27) (0.29) (0.29) 
7 0.09 0.09 0.11 0.08 0.08 0.09 0.02 0.04 0.07 -0.07 -0.09 -0.09 
(0.27) (0.27) (0.29) 
8 0.08 0.09 0.09 0.07 0.08 0.08 0.00 0.02 0.04 = -0.09 -0.09 -0.10 
(0.26) (0.27) (0.28) 
9 0.08 0.08 0.06 0.10 0.10 0.12 -0.09 -0.10 -0.17 -0.13 -0.11 -0.13 
(0.31) (0.31) (0.32) 
10 0.07 0.06 0.04 0.10 0.09 0.09 -0.09 -0.10 -0.17 -0.14 -0.13 -0.09 
(0.30) (0.29) (0.28) 
11 0.07 0.07 0.04 0.09 0.08 0.08  -0.06 -0.07 -0.13 -0.12 -0.11 -0.08 
(0.28) (0.28) (0.26) 
12 0.07 0.06 0.04 0.09 0.09 0.08  -0.10 -0.09 -0.16 -0.10 -0.09 -0.09 


(0.29) (0.28) (0.27) 
English learner 0.12 0.12 0.13 0.12 0.12 0.14 0.01 -0.02 -0.03 0.00 -0.01 0.00 


student (0.32) (0.33) (0.34) 

Special = 0.14 0.14 0.14 016 017 017 -0.06 -0.07 -0.07 0.04 0.03 -0.04 
education (0.37) (0.37) (0.37) 

student 

Fractionofdays 0.09 0.09 0.07 O15 O15 013 -0.27 -0.27  -032 018 018 -014 
absent (0.20) (0.21) (0.19) 

Numberof 014 016 0.07 018 0.20 0.09 -0.05 0.05 -0.07 -0.09 -010 -0.07 
suspensions (0.65) (0.74) (0.43) 


At-risk status 0.42 0.43 0.42 0.42 0.48 0.48 0.00 -0.10 -0.13 0.05 0.03 0.03 
(0.49) (0.50) (0.50) 

Composite of 4.08 4.12 4.14 4.00 4.06 4.04 0.25 0.22 0.36 0.39 0.37 0.35 

covariates (0.32) (0.29) (0.28) 


Note: This table explores nonresponse bias through the mean, standard deviation, and correlation with the scale across covariates. 
Source: Authors’ analyses based on survey and administrative data provided by the District of Columbia Public Schools, 2017/18 to 2019/20. 
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Table B12. Exploration of nonresponse bias for student satisfaction (student survey) 
Mean for original study 


sample (with standard Difference in standard Correlation with student 
Mean for survey sample deviation) deviation units satisfaction scale 
Covariates and 
units 2017/18 2018/19 2019/20 2017/18 2018/19 2019/20 2017/18 2018/19 2019/20 2017/18 2018/19 2019/20 
Female 0.50 0.50 0.51 0.49 0.49 0.49 0.02 0.03 0.03 0.01 -0.02 -0.02 
(0.50) (0.50) (0.50) 
Grade level 
3 0.16 0.15 0.17 0.13 0.13 0.13 0.09 0.07 0.13 0.17 0.17 0.19 
(0.34) (0.34) (0.33) 
4 0.16 0.15 0.17 0.13 0.13 0.12 0.09 0.08 0.13 0.14 0.17 0.13 
(0.34) (0.33) (0.33) 
5 0.15 0.15 0.16 0.12 0.12 0.11 0.09 0.08 0.13 0.10 0.10 0.15 
(0.32) (0.32) (0.32) 
6 0.08 0.10 0.11 0.08 0.09 0.09 0.00 0.02 0.06 0.00 -0.01 -0.06 
(0.27) (0.29) (0.29) 
7 0.09 0.09 0.11 0.08 0.08 0.09 0.03 0.04 0.07 -0.07 -0.11 -0.11 
(0.27) (0.27) (0.29) 
8 0.08 0.09 0.09 0.07 0.08 0.08 0.01 0.02 0.04 = -0.10 -0.10 -0.12 
(0.26) (0.27) (0.28) 
9 0.08 0.08 0.06 0.10 0.10 0.12 -0.09 -0.10 -0.17 -0.08 -0.08 -0.11 
(0.31) (0.31) (0.32) 
10 0.07 0.06 0.04 0.10 0.09 0.09 -0.09 -0.10 -0.17 -0.11 -0.10 -0.09 
(0.30) (0.29) (0.28) 
11 0.07 0.07 0.04 0.09 0.08 0.08  -0.06 -0.07 -0.13 -0.10 -0.10 -0.09 
(0.28) (0.28) (0.26) 
12 0.07 0.06 0.04 0.09 0.09 0.08  -0.10 -0.09 -0.16 -0.10 -0.10 -0.11 


(0.29) (0.28) (0.27) 
English learner 0.12 0.12 0.13 0.12 0.12 0.14 0.01 -0.01 -0.03 0.10 0.11 0.11 


student (0.32) (0.33) (0.34) 

Special = 0.14 0.14 0.14 016 017 0.17 -0.06 -0.06 -0.07 -003 -0.02 -0.02 
education (0.37) (0.37) (0.37) 

student 

Fractionofdays 0.09 0.09 0.07 O15 O15 013 -0.27 -0.27 -032 016 018 0.16 
absent (0.20) (0.21) (0.19) 

Numberof 014 016 0.07 018 0.20 0.09 -0.05 0.05 -0.07 -010 -013 -0.09 
suspensions (0.65) (0.74) (0.43) 


At-risk status 042 043 O42 042 048 O48 +000 -010 -013 -0.06 -0.06 -0.07 
(0.49) (0.50) (0.50) 

Composite of 3.75 3.75 3.76 3.69 369 367 4210.22 O19 032 0.36 0.38 0.38 

covariates (0.28) (0.29) (0.29) 


Note: This table explores nonresponse bias through the mean, standard deviation, and correlation with the scale across covariates. 
Source: Authors’ analyses based on survey and administrative data provided by the District of Columbia Public Schools, 2017/18 to 2019/20. 
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Table B13. Exploration of nonresponse bias for sense of belonging (student survey) 


Mean for survey sample deviation) deviation units belonging scale 
Covariates and 
units 2017/18 2018/19 2019/20 2017/18 2018/19 2019/20 2017/18 2018/19 2019/20 2017/18 2018/19 2019/20 
Female 0.50 0.50 0.51 0.49 0.49 0.49 0.02 0.02 0.03 -0.01 -0.04 -0.05 
(0.50) (0.50) (0.50) 
Grade level 
3 0.16 0.15 0.17 0.13 0.13 0.13 0.09 0.07 0.13 0.16 0.18 0.16 
(0.34) (0.34) (0.33) 
4 0.16 0.15 0.17 0.13 0.13 0.12 0.08 0.08 0.13 0.14 0.16 0.14 
(0.34) (0.33) (0.33) 
5 0.15 0.14 0.16 0.12 0.12 0.11 0.09 0.08 0.13 0.13 0.14 0.17 
(0.32) (0.32) (0.32) 
6 0.08 0.10 0.11 0.08 0.09 0.09 0.01 0.02 0.06 -0.03 -0.04 -0.09 
(0.27) (0.29) (0.29) 
vi 0.09 0.09 0.11 0.08 0.08 0.09 0.03 0.04 0.07 -0.08 -0.10 -0.12 
(0.27) (0.27) (0.29) 
8 0.08 0.09 0.09 0.07 0.08 0.08 0.01 0.03 0.04 -0.09 -0.10 -0.10 
(0.26) (0.27) (0.28) 
9 0.08 0.08 0.06 0.10 0.10 0.12 -0.09 -0.10 -0.17 -0.11 -0.12 -0.13 
(0.31) (0.31) (0.32) 
10 0.07 0.06 0.04 0.10 0.09 0.09 -0.09 -0.10 -0.17 -0.12 -0.11 -0.09 
(0.30) (0.29) (0.28) 
11 0.07 0.07 0.04 0.09 0.08 0.08 -0.05 -0.07 -0.13 -0.09 -0.09 -0.08 
(0.28) (0.28) (0.26) 
12 0.07 0.06 0.04 0.09 0.09 0.08 -0.10 -0.08 -0.16 -0.07 -0.07 -0.08 
(0.29) (0.28) (0.27) 
English learner 0.12 0.12 0.13 0.12 0.12 0.14 0.01 -0.01 -0.03 0.05 0.05 0.06 
student (0.32) (0.33) (0.34) 
Special 0.14 0.14 0.14 0.16 0.17 0.17 -0.06 -0.06 -0.07 -0.01 0.01 0.00 
education (0.37) (0.37) (0.37) 
student 
Fraction of days 0.09 0.09 0.07 0.15 0.15 0.13 -0.26 -0.27 -0.32 -0.13 -0.15 -0.14 
absent (0.20) (0.21) (0.19) 
Number of 0.14 0.16 0.07 0.18 0.20 0.09 -0.05 -0.05 -0.07 -0.05 -0.07 -0.05 
suspensions (0.65) (0.74) (0.43) 
At-risk status 0.42 0.43 0.42 0.42 0.48 0.48 0.00 -0.10 -0.13 0.01 0.01 -0.01 
(0.49) (0.50) (0.50) 
Composite of 3.65 3.65 3.66 3.58 3.59 3.55 0.22 0.19 0.32 0.33 0.35 0.36 
covariates (0.28) (0.30) (0.32) 
Note: This table explores nonresponse bias through the mean, standard deviation, and correlation with the scale across covariates. 
Source: Authors’ analyses based on survey and administrative data provided by the District of Columbia Public Schools, 2017/18 to 2019/20. 
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Mean for original study 
sample (with standard 


Difference in standard 


Correlation with sense of 


Table B14. Exploration of nonresponse bias for changes in perseverance (student survey) 
Mean for original study 


sample (with standard Difference in standard Correlation with changes 
Mean for survey sample deviation) deviation units in perseverance scale 
2017/18 2018/19 2017/18 2017/18 2018/19 2017/18 2017/18 2018/19 2017/18 2017/18 2018/19 2017/18 
Covariates and & & & & & & & & & & & & 
units 2018/19 2019/20 2019/20 2018/19 2019/20 2019/20 2018/19 2019/20 2019/20 2018/19 2019/20 2019/20 
Female 0.50 0.51 0.51 0.49 0.49 0.49 0.03 0.04 0.05 -0.03 -0.04 -0.09 
(0.50) (0.50) (0.50) 
Grade level 
3 0.21 0.22 0.27 0.15 0.15 0.17 0.16 0.19 0.28 0.01 0.01 0.04 
(0.36) (0.36) (0.37) 
4 0.20 0.21 0.19 0.14 0.14 0.14 0.15 0.21 0.16 0.02 -0.01 -0.03 
(0.35) (0.35) (0.34) 
5 0.14 0.15 0.20 0.11 0.11 0.13 0.09 0.12 0.19 -0.04 -0.04 -0.07 
(0.31) (0.31) (0.34) 
6 0.10 0.14 0.14 0.09 0.11 0.11 0.03 0.09 0.09 -0.04 -0.03 -0.02 
(0.29) (0.31) (0.31) 
7 0.11 0.12 0.06 0.10 0.09 0.10 0.04 0.09 -0.13 0.00 0.03 0.02 
(0.30) (0.29) (0.30) 
8 0.05 0.04 0.04 0.07 0.08 0.08 -0.06 -0.14 -0.17 0.00 0.01 0.01 
(0.25) (0.27) (0.28) 
9 0.06 0.04 0.06 0.11 0.11 0.12 -0.16 -0.22 -0.20 0.00 0.00 0.05 
(0.31) (0.31) (0.33) 
10 0.07 0.04 0.04 0.11 0.10 0.13 -0.14 -0.19 -0.25 0.03 0.04 0.06 
(0.31) (0.30) (0.33) 
11 0.06 0.04 0.00 0.10 0.10 0.02 -0.14 -0.21 -0.13 0.03 0.03 0.01 
(0.31) (0.30) (0.13) 
12 0.00 0.00 0.00 0.01 0.01 0.00 -0.09 -0.09 -0.06 0.00 -0.01 0.00 


(0.11) (0.10) (0.06) 
English learner 0.13 0.11 0.11 0.13 0.12 0.13 0.00 -0.05 -0.04 0.03 -0.03 -0.02 


student (0.33) (0.33) (0.33) 

Special § 0.12 013 013 0.16 0.17 40.16 -010 -010 -0.09 0.02 0.02 0.05 
education (0.37) (0.37) (0.37) 

student 

Fraction ofdays 0.07 0.06 0.06 0.12 0.12 0.10 -031 -035 -032 0.02 0.04 0.07 
absent (0.17) (0.18) (0.15) 

Numberof 010 O11 0.09 0.18 0.20 018 -012 -012 -014 0.03 0.01 0.05 
suspensions (0.66) (0.75) (0.67) 


At-risk status 040 039 O40 043 O47 42043 -0.05 -015 -006 001 001 #001 
(0.49) (0.50) (0.50) 

Compositeof -0.05 -0.08 -0.15 -0.03 -0.04 -0.07 -0.20 -031 -040 0.09 010 0.17 

covariates (0.08) (0.10) (0.20) 


Note: This table explores nonresponse bias through the mean, standard deviation, and correlation with the scale across covariates. 
Source: Authors’ analyses based on survey and administrative data provided by the District of Columbia Public Schools, 2017/18 to 2019/20. 
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Table B15. Exploration of nonresponse bias for changes in self-management (student survey) 
Mean for original study 


sample (with standard Difference in standard Correlation with changes 
Mean for survey sample deviation) deviation units in self-management scale 
2017/18 2018/19 2017/18 2017/18 2018/19 2017/18 2017/18 2018/19 2017/18 2017/18 2018/19 2017/18 
Covariates and & & & & & & & & & & & & 
units 2018/19 2019/20 2019/20 2018/19 2019/20 2019/20 2018/19 2019/20 2019/20 2018/19 2019/20 2019/20 
Female 0.50 0.51 0.51 0.49 0.49 0.49 0.03 0.04 0.05 -0.02 -0.04 -0.07 
(0.50) (0.50) (0.50) 
Grade level 
3 0.21 0.22 0.28 0.15 0.15 0.17 0.18 0.20 0.29 0.03 0.02 0.03 
(0.36) (0.36) (0.37) 
4 0.20 0.22 0.19 0.14 0.14 0.14 0.16 0.22 0.16 0.01 -0.02 -0.03 
(0.35) (0.35) (0.34) 
5 0.14 0.15 0.20 0.11 0.11 0.13 0.08 0.12 0.19 -0.01 -0.03 -0.05 
(0.31) (0.31) (0.34) 
6 0.10 0.13 0.14 0.09 0.11 0.11 0.02 0.09 0.08 -0.04 -0.04 -0.01 
(0.29) (0.31) (0.31) 
7 0.11 0.12 0.06 0.10 0.09 0.10 0.04 0.08 -0.13 -0.01 0.04 0.01 
(0.30) (0.29) (0.30) 
8 0.05 0.04 0.04 0.07 0.08 0.08 -0.07 -0.14 -0.17 0.00 0.00 0.02 
(0.25) (0.27) (0.28) 
9 0.06 0.04 0.06 0.11 0.11 0.12 -0.16 -0.23 -0.20 -0.01 0.02 0.02 
(0.31) (0.31) (0.33) 
10 0.07 0.04 0.04 0.11 0.10 0.13 -0.14 -0.19 -0.25 0.02 0.02 0.04 
(0.31) (0.30) (0.33) 
11 0.06 0.04 0.00 0.10 0.10 0.02 -0.14 -0.21 -0.13 -0.01 0.01 0.02 
(0.31) (0.30) (0.13) 
12 0.00 0.00 0.00 0.01 0.01 0.00 -0.09 -0.09 -0.06 -0.01 0.00 0.01 


(0.11) (0.10) (0.06) 
English learner 0.12 0.11 0.11 0.13 0.12 0.13 -0.01 -0.05 -0.05 0.01 0.01 -0.03 


student (0.33) (0.33) (0.33) 

Special  § 012 0.13 013 0.16 0.17 0.16 -010 -010 -009 0.02 0.01 0.03 
education (0.37) (0.37) (0.37) 

student 

Fraction ofdays 0.07 0.06 0.06 0.12 0.12 0.10 -031 -036 -032 0.01 0.02 0.07 
absent (0.17) (0.18) (0.15) 

Numberof 010 O11 0.09 0.18 0.20 018 -012 -012 -014 0.03 0.02 0.06 
suspensions (0.66) (0.75) (0.67) 


At-risk status 0.40 039 O40 043 047 «2043 005 015 -006 001 -001 0.01 
(0.49) (0.50) (0.50) 

Compositeof 0.01 -0.05 -0.05 0.01 -0.03 0.02 -0.06 -0.30 -0.36 0.07 0.08 0.13 

covariates (0.06) (0.07) (0.17) 


Note: This table explores nonresponse bias through the mean, standard deviation, and correlation with the scale across covariates. 
Source: Authors’ analyses based on survey and administrative data provided by the District of Columbia Public Schools, 2017/18 to 2019/20. 
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Table B16. Exploration of nonresponse bias for changes in self-efficacy (student survey) 
Mean for original study 


sample (with standard Difference in standard Correlation with changes 
Mean for survey sample deviation) deviation units in self-efficacy scale 
2017/18 2018/19 2017/18 2017/18 2018/19 2017/18 2017/18 2018/19 2017/18 2017/18 2018/19 2017/18 
Covariates and & & & & & & & & & & & & 
units 2018/19 2019/20 2019/20 2018/19 2019/20 2019/20 2018/19 2019/20 2019/20 2018/19 2019/20 2019/20 
Female 0.50 0.51 0.51 0.49 0.49 0.49 0.03 0.04 0.05 -0.03 -0.04 -0.07 
(0.50) (0.50) (0.50) 
Grade level 
3 0.21 0.22 0.28 0.15 0.15 0.17 0.17 0.19 0.29 0.03 0.03 0.09 
(0.36) (0.36) (0.37) 
4 0.20 0.21 0.19 0.14 0.14 0.14 0.16 0.22 0.16 0.02 0.01 -0.04 
(0.35) (0.35) (0.34) 
5 0.14 0.15 0.20 0.11 0.11 0.13 0.08 0.12 0.19 -0.05 -0.07 -0.10 
(0.31) (0.31) (0.34) 
6 0.10 0.14 0.14 0.09 0.11 0.11 0.03 0.09 0.09 -0.05 -0.04 0.00 
(0.29) (0.31) (0.31) 
7 0.11 0.12 0.06 0.10 0.09 0.10 0.04 0.09 -0.13 0.00 0.03 -0.01 
(0.30) (0.29) (0.30) 
8 0.05 0.04 0.04 0.07 0.08 0.08 -0.07 -0.14 -0.17 -0.03 -0.04 0.01 
(0.25) (0.27) (0.28) 
9 0.06 0.04 0.06 0.11 0.11 0.12 -0.16 -0.23 -0.20 0.00 0.01 0.05 
(0.31) (0.31) (0.33) 
10 0.07 0.04 0.04 0.11 0.10 0.13 -0.14 -0.19 -0.25 0.03 0.03 0.05 
(0.31) (0.30) (0.33) 
11 0.06 0.04 0.00 0.10 0.10 0.02 -0.14 -0.21 -0.13 0.04 0.04 0.02 
(0.31) (0.30) (0.13) 
12 0.00 0.00 0.00 0.01 0.01 0.00 -0.09 -0.09 -0.06 0.02 -0.01 0.01 


(0.11) (0.10) (0.06) 
English learner 0.12 0.11 0.11 0.13 0.12 0.13 -0.01 -0.06 -0.05 0.01 -0.02 -0.03 


student (0.33) (0.33) (0.33) 

Special = 0.12 013 013 0.16 0.17 0.16 -O11 -010 -009 0.02 0.02 0.03 
education (0.37) (0.37) (0.37) 

student 

Fraction ofdays 0.07 0.06 0.06 0.12 0.12 0.10 -031 -036 -032 0.04 0.04 0.06 
absent (0.17) (0.18) (0.15) 

Numberof 010 O11 0.09 0.18 0.20 018 -012 -012 -014 0.03 0.01 0.05 
suspensions (0.66) (0.75) (0.67) 


At-risk status 0.40 0.39 0.40 0.43 0.47 0.43 -0.05 -0.16 -0.06 0.00 0.01 0.00 
(0.49) (0.50) (0.50) 

Composite of -0.05 -0.09 -0.17 = -0.03 -0.06 -0.11 -0.20 = -0.20 -0.31 0.11 0.12 0.18 

covariates (0.11) (0.12) (0.20) 


Note: This table explores nonresponse bias through the mean, standard deviation, and correlation with the scale across covariates. 
Source: Authors’ analyses based on survey and administrative data provided by the District of Columbia Public Schools, 2017/18 to 2019/20. 
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Table B17. Exploration of nonresponse bias for changes in social awareness (student survey) 
Mean for original study 


sample (with standard Difference in standard Correlation with changes 
Mean for survey sample deviation) deviation units in social awareness scale 
2017/18 2018/19 2017/18 2017/18 2018/19 2017/18 2017/18 2018/19 2017/18 2017/18 2018/19 2017/18 
Covariates and & & & & & & & & & & & & 
units 2018/19 2019/20 2019/20 2018/19 2019/20 2019/20 2018/19 2019/20 2019/20 2018/19 2019/20 2019/20 
Female 0.50 0.52 0.51 0.49 0.49 0.49 0.03 0.04 0.05 -0.03 -0.01 -0.06 
(0.50) (0.50) (0.50) 
Grade level 
3 0.21 0.22 0.28 0.15 0.15 0.17 0.17 0.20 0.29 0.01 -0.01 0.03 
(0.36) (0.36) (0.37) 
4 0.20 0.21 0.19 0.14 0.14 0.14 0.16 0.22 0.16 0.00 0.00 -0.08 
(0.35) (0.35) (0.34) 
5 0.14 0.15 0.20 0.11 0.11 0.13 0.08 0.12 0.19 -0.06 -0.07 -0.08 
(0.31) (0.31) (0.34) 
6 0.10 0.14 0.14 0.09 0.11 0.11 0.03 0.09 0.08 -0.03 -0.03 0.00 
(0.29) (0.31) (0.31) 
7 0.11 0.12 0.06 0.10 0.09 0.10 0.04 0.09 -0.13 0.00 0.04 0.04 
(0.30) (0.29) (0.30) 
8 0.05 0.04 0.04 0.07 0.08 0.08 -0.07 -0.14 -0.17 0.00 0.03 0.03 
(0.25) (0.27) (0.28) 
9 0.06 0.04 0.06 0.11 0.11 0.12 -0.16 -0.23 -0.20 0.02 0.03 0.08 
(0.31) (0.31) (0.33) 
10 0.07 0.04 0.04 0.11 0.10 0.13 -0.14 -0.19 -0.25 0.05 0.05 0.09 
(0.31) (0.30) (0.33) 
11 0.06 0.04 0.00 0.10 0.10 0.02 -0.14 -0.21 -0.13 0.04 0.03 -0.01 
(0.31) (0.30) (0.13) 
12 0.00 0.00 0.00 0.01 0.01 0.00 -0.09 -0.09 -0.06 0.01 0.01 -0.01 


(0.11) (0.10) (0.06) 
English learner 0.12 0.11 0.11 0.13 0.12 0.13 -0.01 -0.05 -0.05 -0.01 0.00 -0.03 


student (0.33) (0.33) (0.33) 

Special = 0.12 013 013 0.16 0.17 016 -O11 -010 -010 002 001 0.01 
education (0.37) (0.37) (0.37) 

student 

Fraction ofdays 0.07 0.06 0.06 0.12 0.12 0.10 -031 -036 -032 0.02 0.04 0.06 
absent (0.17) (0.18) (0.15) 

Numberof 010 O11 0.09 0.18 0.20 018 -012 -012 -014 0.04 0.01 0.04 
suspensions (0.66) (0.75) (0.67) 


At-risk status 0.40 039 O40 043 047 «2043 -005 015 -006 -001 -002 -0.03 
(0.49) (0.50) (0.50) 

Composite of  -0.03 -0.07 -0.13 -0.02 -0.03 -005 -0.18 -040 -039 0.11 0.12 0.19 

covariates (0.09) (0.10) (0.21) 


Note: This table explores nonresponse bias through the mean, standard deviation, and correlation with the scale across covariates. 
Source: Authors’ analyses based on survey and administrative data provided by the District of Columbia Public Schools, 2017/18 to 2019/20. 
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Table B18. Exploration of nonresponse bias for changes in rigorous expectations (student survey) 


Mean for original study Correlation with changes 
sample (with standard Difference in standard in rigorous expectations 
Mean for survey sample deviation) deviation units Ler] (=) 
2017/18 2018/19 2017/18 2017/18 2018/19 2017/18 2017/18 2018/19 2017/18 2017/18 2018/19 2017/18 
Covariates and & & & & & & & & & & & & 
units 2018/19 2019/20 2019/20 2018/19 2019/20 2019/20 2018/19 2019/20 2019/20 2018/19 2019/20 2019/20 
Female 0.50 0.51 0.51 0.49 0.49 0.49 0.03 0.04 0.05 -0.04 0.00 -0.06 
(0.50) (0.50) (0.50) 
Grade level 
3 0.21 0.22 0.28 0.15 0.15 0.17 0.16 0.19 0.29 0.04 0.06 0.13 
(0.36) (0.36) (0.37) 
4 0.20 0.21 0.19 0.14 0.14 0.14 0.16 0.21 0.16 0.05 0.03 -0.09 
(0.35) (0.35) (0.34) 
5 0.14 0.15 0.20 0.11 0.11 0.13 0.09 0.12 0.19 -0.12 -0.15 -0.15 
(0.31) (0.31) (0.34) 
6 0.10 0.14 0.14 0.09 0.11 0.11 0.03 0.09 0.09 -0.03 -0.03 0.00 
(0.29) (0.31) (0.31) 
7 0.11 0.12 0.06 0.10 0.09 0.10 0.04 0.09 -0.13 0.02 0.02 0.00 
(0.30) (0.29) (0.30) 
8 0.05 0.04 0.04 0.07 0.08 0.08 -0.06 -0.14 -0.17 0.00 -0.01 0.02 
(0.25) (0.27) (0.28) 
9 0.06 0.04 0.06 0.11 0.11 0.12 -0.16 -0.22 -0.20 0.01 0.01 0.06 
(0.31) (0.31) (0.33) 
10 0.07 0.04 0.04 0.11 0.10 0.13 -0.14 -0.19 -0.25 0.03 0.05 0.07 
(0.31) (0.30) (0.33) 
11 0.06 0.04 0.00 0.10 0.10 0.02 -0.14 -0.20 -0.13 0.02 0.04 0.04 
(0.31) (0.30) (0.13) 
12 0.00 0.00 0.00 0.01 0.01 0.00 -0.09 -0.09 -0.06 0.00 0.00 0.00 


(0.11) (0.10) (0.06) 
English learner 0.13 0.11 0.11 0.13 0.12 0.13 -0.01 -0.05 -0.05 0.01 -0.01 -0.01 


student (0.33) (0.33) (0.33) 

Special 013 0.13 013 0.16 0.17 0.16 -010 -009 -009 0.02 0.01 0.03 
education (0.37) (0.37) (0.37) 

student 

Fractionofdays 0.07 0.06 0.06 0.12 0.12 0.10 -030 -035 -032 0.03 0.05 0.07 
absent (0.17) (0.18) (0.15) 

Numberof 010 O11 0.09 0.18 0.20 018 -012 -012 -014 0.01 0.03 0.03 
suspensions (0.66) (0.75) (0.67) 


At-risk status 0.40 0.39 0.40 0.43 0.47 0.43 = -0.05 -0.15 -0.06 -0.01 0.00 = -0.01 
(0.49) (0.50) (0.50) 

Composite of -0.02 -0.07 -0.14 0.00 -0.03 -0.05 -0.15 -0.24 -0.34 0.14 0.17 0.23 

covariates (0.11) (0.14) (0.26) 


Note: This table explores nonresponse bias through the mean, standard deviation, and correlation with the scale across covariates. 
Source: Authors’ analyses based on survey and administrative data provided by the District of Columbia Public Schools, 2017/18 to 2019/20. 
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Table B19. Exploration of nonresponse bias for changes in student satisfaction (student survey) 


Mean for original study Correlation with changes 
sample (with standard Difference in standard in student satisfaction 
Mean for survey sample deviation) deviation units Ler] (=) 
2017/18 2018/19 2017/18 2017/18 2018/19 2017/18 2017/18 2018/19 2017/18 2017/18 2018/19 2017/18 
Covariates and & & & & & & & & & & & & 
units 2018/19 2019/20 2019/20 2018/19 2019/20 2019/20 2018/19 2019/20 2019/20 2018/19 2019/20 2019/20 
Female 0.50 0.51 0.51 0.49 0.49 0.49 0.03 0.04 0.04 -0.03 -0.02 -0.08 
(0.50) (0.50) (0.50) 
Grade level 
3 0.21 0.22 0.27 0.15 0.15 0.17 0.16 0.19 0.28 0.06 0.03 0.13 
(0.36) (0.36) (0.37) 
4 0.20 0.21 0.19 0.14 0.14 0.14 0.15 0.21 0.15 0.03 0.05 -0.08 
(0.35) (0.35) (0.34) 
5 0.14 0.15 0.20 0.11 0.11 0.13 0.08 0.12 0.19 -0.07 -0.10 -0.12 
(0.31) (0.31) (0.34) 
6 0.10 0.14 0.14 0.09 0.11 0.11 0.04 0.09 0.09 -0.07 -0.06 -0.04 
(0.29) (0.31) (0.31) 
7 0.11 0.12 0.06 0.10 0.09 0.10 0.05 0.09 -0.13 0.01 0.03 0.04 
(0.30) (0.29) (0.30) 
8 0.05 0.04 0.04 0.07 0.08 0.08 -0.06 -0.14 -0.17 0.02 0.04 0.05 
(0.25) (0.27) (0.28) 
9 0.06 0.04 0.06 0.11 0.11 0.12 -0.16 -0.22 -0.20 -0.02 -0.01 0.01 
(0.31) (0.31) (0.33) 
10 0.07 0.04 0.04 0.11 0.10 0.13 -0.14 -0.19 -0.25 0.02 0.04 0.06 
(0.31) (0.30) (0.33) 
11 0.06 0.04 0.00 0.10 0.10 0.02 -0.14 -0.20 -0.13 0.01 0.02 0.02 
(0.31) (0.30) (0.13) 
12 0.00 0.00 0.00 0.01 0.01 0.00 -0.09 -0.09 -0.06 0.03 0.01 0.00 


(0.11) (0.10) (0.06) 
English learner 0.13 0.11 0.11 0.13 0.12 0.13 -0.01 -0.05 -0.05 0.02 -0.01 -0.01 


student (0.33) (0.33) (0.33) 

Special 013 013 013 0.16 0.17 0.16 -0.09 -009 -0.09 0.02 0.02 0.06 
education (0.37) (0.37) (0.37) 

student 

Fraction ofdays 0.07 0.06 0.06 0.12 0.12 0.10 -030 -035 -032 0.04 0.04 0.08 
absent (0.17) (0.18) (0.15) 

‘Numberof 010 O11 0.09 0.18 0.20 018 -012 -012 -014 0.01 0.02 0.03 
suspensions (0.66) (0.75) (0.67) 


At-risk status 0.40 039 040 043 047 0.43 -0.05 -0.15 -0.06 0.03 0.01 0.04 
(0.49) (0.50) (0.50) 

Compositeof  -0.09 -0.13  -0.24 -0.07 -010 -016 -0.16 -0.25 -0.35 0.13 0.15 0.23 

covariates (0.11) (0.12) (0.22) 


Note: This table explores nonresponse bias through the mean, standard deviation, and correlation with the scale across covariates. 
Source: Authors’ analyses based on survey and administrative data provided by the District of Columbia Public Schools, 2017/18 to 2019/20. 
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Table B20. Exploration of nonresponse bias for changes in sense of belonging (student survey) 
Mean for original study 


sample (with standard Difference in standard Correlation with changes 
Mean for survey sample deviation) deviation units in sense of belonging scale 
2017/18 2018/19 2017/18 2017/18 2018/19 2017/18 2017/18 2018/19 2017/18 2017/18 2018/19 2017/18 
Covariates and & & & & & & & & & & & & 
units 2018/19 2019/20 2019/20 2018/19 2019/20 2019/20 2018/19 2019/20 2019/20 2018/19 2019/20 2019/20 
Female 0.50 0.51 0.51 0.49 0.49 0.49 0.03 0.03 0.04 -0.05 -0.04 -0.10 
(0.50) (0.50) (0.50) 
Grade level 
3 0.21 0.22 0.27 0.15 0.15 0.17 0.16 0.19 0.28 0.03 0.03 0.15 
(0.36) (0.36) (0.37) 
4 0.19 0.21 0.19 0.14 0.14 0.14 0.14 0.20 0.15 0.06 0.07 -0.13 
(0.35) (0.35) (0.34) 
5 0.14 0.15 0.20 0.11 0.11 0.13 0.08 0.12 0.19 -0.14 -0.19 -0.16 
(0.31) (0.31) (0.34) 
6 0.11 0.14 0.14 0.09 0.11 0.11 0.04 0.10 0.10 -0.03 -0.03 0.03 
(0.29) (0.31) (0.31) 
7 0.11 0.12 0.06 0.10 0.09 0.10 0.05 0.09 -0.13 0.02 0.05 0.01 
(0.30) (0.29) (0.30) 
8 0.05 0.04 0.04 0.07 0.08 0.08 -0.05 -0.13 -0.17 -0.02 -0.02 0.01 
(0.25) (0.27) (0.28) 
9 0.06 0.04 0.06 0.11 0.11 0.12 -0.16 -0.22 -0.19 0.01 0.02 0.06 
(0.31) (0.31) (0.33) 
10 0.07 0.04 0.04 0.11 0.10 0.13 -0.14 -0.19 -0.25 0.04 0.05 0.08 
(0.31) (0.30) (0.33) 
11 0.06 0.04 0.00 0.10 0.10 0.02 -0.14 -0.20 -0.13 0.03 0.04 0.01 
(0.31) (0.30) (0.13) 
12 0.00 0.00 0.00 0.01 0.01 0.00 -0.09 -0.09 -0.06 0.02 0.02 0.01 


(0.11) (0.10) (0.06) 
English learner 0.13 0.11 0.11 0.13 0.12 0.13 -0.01 -0.05 -0.05 0.01 -0.02 -0.02 


student (0.33) (0.33) (0.33) 

Special 013 013 013 0.16 0.17 0.16 -010 -009 -0.09 0.02 0.02 0.05 
education (0.37) (0.37) (0.37) 

student 

Fraction ofdays 0.07 0.06 0.06 0.12 0.12 0.10 -030 -035 -0.32 0.04 0.03 0.07 
absent (0.17) (0.18) (0.15) 

Numberof 010 O11 0.09 0.18 0.20 018 -012 -012 -014 0.01 0.02 0.04 
suspensions (0.66) (0.75) (0.67) 


At-risk status 0.40 0.40 0.40 0.43 0.47 0.43 -0.05 -0.15 -0.06 0.02 0.00 0.02 
(0.49) (0.50) (0.50) 

Composite of -0.06 -0.11 -0.21 -0.03 -0.08 -0.13 -0.16 -0.16 -0.30 0.17 0.21 0.28 

covariates (0.15) (0.17) (0.28) 


Note: This table explores nonresponse bias through the mean, standard deviation, and correlation with the scale across covariates. 
Source: Authors’ analyses based on survey and administrative data provided by the District of Columbia Public Schools, 2017/18 to 2019/20. 
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Table B21. Exploration of nonresponse bias for perseverance (teacher survey) 
Mean for original study 


sample (with standard _ Difference in standard Correlation with 
Mean for survey sample deviation) deviation units perseverance scale 
Covariates and units 2017/18 2018/19 2017/18 2018/19 2017/18 2018/19 2017/18 2018/19 
Female 0.79 0.77 0.77 0.75 0.05 0.05 0.05 0.06 
(0.42) (0.43) 
Teacher’s years of experience 
0-1 years of 0.24 0.21 0.25 0.21 -0.01 -0.02 -0.07 -0.09 
experience (0.43) (0.41) 
2-3 years of 0.23 0.21 0.22 0.20 0.01 0.00 -0.03 -0.02 
experience (0.42) (0.40) 
4-5 years of 0.11 0.16 0.12 0.15 -0.01 0.01 -0.02 -0.01 
experience (0.32) (0.36) 
6-10 years of 0.17 0.18 0.17 0.18 0.01 0.01 0.02 0.01 
experience (0.38) (0.38) 
More than 10 years 0.24 0.25 0.24 0.25 0.00 0.00 0.10 0.09 
of experience (0.43) (0.43) 
Teacher’s race/ethnicity 
Black 0.50 0.48 0.54 0.50 -0.08 -0.05 0.04 0.00 
(0.50) (0.50) 
Hispanic 0.07 0.08 0.07 0.07 0.01 0.01 0.06 0.06 
(0.26) (0.26) 
White 0.32 0.34 0.27 0.31 0.10 0.05 -0.10 -0.05 
(0.45) (0.46) 
Other 0.04 0.04 0.04 0.04 0.01 0.00 0.03 0.02 
(0.19) (0.19) 
Not reported 0.07 0.07 0.08 0.07 -0.03 0.00 0.02 0.01 
(0.26) (0.25) 
Type of school 
Elementary 0.72 0.68 0.70 0.67 0.05 0.03 0.25 0.22 
(0.46) (0.47) 
Middle 0.29 0.32 0.30 0.32 -0.01 -0.01 -0.06 -0.09 
(0.46) (0.47) 
High 0.20 0.22 0.21 0.23 -0.04 -0.03 -0.17 -0.16 
(0.41) (0.42) 
Nontraditional 0.00 0.00 0.00 0.00 0.00 0.00 -0.02 0.01 
(0.06) (0.05) 
School’s STAR rating 
1 0.13 0.14 0.14 0.14 -0.01 -0.01 -0.15 -0.23 
(0.34) (0.35) 
2 0.23 0.22 0.23 0.21 -0.02 0.03 -0.12 -0.11 
(0.42) (0.41) 
3 0.35 0.19 0.34 0.20 0.02 -0.02 0.07 -0.01 
(0.47) (0.40) 
4 0.16 0.30 0.17 0.30 -0.02 -0.01 0.08 0.17 
(0.37) (0.46) 
5 0.14 0.16 0.13 0.15 0.02 0.02 0.12 0.14 
(0.33) (0.36) 
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Table B21. Exploration of nonresponse bias for perseverance (teacher survey) (continued) 


Mean for original study 
sample (with standard _ Difference in standard Correlation with 


Mean for survey sample deviation) deviation units perseverance scale 


Covariates and units 2017/18 2018/19 2017/18 2018/19 2017/18 2018/19 2017/18 2018/19 
School’s ward 


1 0.12 0.13 0.12 0.12 0.00 0.02 0.01 -0.03 
(0.32) (0.33) 
2 0.06 0.06 0.05 0.06 0.04 0.01 0.05 0.06 
(0.23) (0.23) 
3 0.13 0.14 0.12 0.13 0.02 0.02 0.06 0.10 
(0.33) (0.34) 
4 0.18 0.16 0.18 0.18 0.01 -0.04 0.05 0.05 
(0.38) (0.38) 
5 0.08 0.09 0.09 0.09 -0.03 0.00 -0.06 -0.05 
(0.29) (0.29) 
6 0.14 0.15 0.16 0.15 -0.05 -0.01 -0.01 0.06 
(0.37) (0.36) 
7 0.12 0.11 0.12 0.12 -0.02 -0.02 -0.01 -0.03 
(0.33) (0.32) 
8 0.17 0.16 0.15 0.15 0.04 0.03 -0.08 -0.15 
(0.36) (0.36) 
Racial/ethnic composition of the school (fraction of students in the school) 
Black 0.63 0.62 0.65 0.62 -0.04 -0.02 -0.16 -0.19 
(0.32) (0.32) 
Hispanic 0.22 0.22 0.21 0.22 0.03 -0.01 0.08 0.07 
(0.25) (0.24) 
White 0.11 0.12 0.10 0.12 0.02 0.03 0.12 0.18 
(0.19) (0.20) 
Other 0.04 0.04 0.04 0.04 0.04 0.04 0.14 0.20 
(0.05) (0.05) 
School-level averages across students 
ELA achievement test -0.09 -0.07 -0.10 -0.07 0.01 0.01 0.20 0.26 
score (0.56) (0.57) 
Math achievement -0.07 -0.05 -0.08 -0.06 0.02 0.01 0.20 0.26 
test score (0.54) (0.54) 
Fraction of days 0.89 0.89 0.89 0.89 0.01 0.01 0.22 0.25 
absent (0.11) (0.12) 
Number of 0.12 0.13 0.13 0.13 -0.03 -0.02 -0.24 -0.26 
suspensions (0.15) (0.18) 
Composite of 3.20 3.21 3.19 3.21 0.01 0.00 0.37 0.40 
covariates (0.29) (0.31) 


ELA is English language arts. STAR is School Transparency and Reporting Framework. 
Note: This table explores nonresponse bias through the mean, standard deviation, and correlation with the scale across covariates. 
Source: Authors’ analyses based on survey and administrative data provided by the District of Columbia Public Schools, 2017/18 to 2018/19. 
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Table B22. Exploration of nonresponse bias for rigorous expectations (teacher survey) 


Mean for original study Correlation with 
sample (with standard  Differenceinstandard rigorous expectations 
Mean for survey sample deviation) deviation units scale 
Covariates and units 2017/18 2018/19 2017/18 2018/19 2017/18 2018/19 2017/18 2018/19 
Female 0.79 0.77 0.77 0.75 0.05 0.05 0.10 0.10 
(0.42) (0.43) 
Teacher’s years of experience 
0-1 years of 0.24 0.20 0.25 0.21 -0.01 -0.02 -0.11 -0.09 
experience (0.43) (0.41) 
2-3 years of 0.23 0.20 0.22 0.20 0.02 0.00 -0.01 -0.02 
experience (0.42) (0.40) 
4-5 years of 0.11 0.16 0.12 0.15 -0.01 0.01 0.02 0.01 
experience (0.32) (0.36) 
6-10 years of 0.17 0.18 0.17 0.18 0.00 0.01 0.02 0.01 
experience (0.38) (0.38) 
More than 10 years 0.24 0.25 0.24 0.25 0.00 0.00 0.09 0.08 
of experience (0.43) (0.43) 
Teacher’s race/ethnicity 
Black 0.50 0.48 0.54 0.50 -0.08 -0.05 0.11 0.10 
(0.50) (0.50) 
Hispanic 0.07 0.08 0.07 0.07 0.01 0.01 -0.01 0.03 
(0.26) (0.26) 
White 0.32 0.34 0.27 0.31 0.10 0.05 -0.11 -0.13 
(0.45) (0.46) 
Other 0.04 0.04 0.04 0.04 0.01 0.00 -0.04 -0.02 
(0.19) (0.19) 
Not reported 0.07 0.07 0.08 0.07 -0.03 0.00 0.03 0.01 
(0.26) (0.25) 
Type of school 
Elementary 0.72 0.68 0.70 0.67 0.05 0.03 0.10 0.12 
(0.46) (0.47) 
Middle 0.29 0.32 0.30 0.32 -0.01 -0.01 -0.02 0.01 
(0.46) (0.47) 
High 0.20 0.22 0.21 0.23 -0.04 -0.02 -0.09 -0.13 
(0.41) (0.42) 
Nontraditional 0.00 0.00 0.00 0.00 0.00 0.00 -0.02 0.00 
(0.06) (0.05) 
School’s STAR rating 
1 0.13 0.14 0.14 0.14 -0.02 -0.01 -0.03 -0.08 
(0.34) (0.35) 
2 0.23 0.22 0.23 0.21 -0.02 0.03 -0.01 0.00 
(0.42) (0.41) 
3 0.35 0.19 0.34 0.20 0.02 -0.03 0.03 0.02 
(0.47) (0.40) 
4 0.16 0.30 0.17 0.30 -0.02 -0.02 -0.03 0.05 
(0.37) (0.46) 
5 0.14 0.16 0.13 0.15 0.03 0.03 0.03 -0.01 
(0.33) (0.36) 
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Table B22. Exploration of nonresponse bias for rigorous expectations (teacher survey) (continued) 


Mean for original study Correlation with 
sample (with standard Differenceinstandard rigorous expectations 


Mean for survey sample deviation) deviation units scale 
Covariates and units 2017/18 2018/19 2017/18 2018/19 2017/18 2018/19 2017/18 2018/19 
School’s ward 


1 0.12 0.13 0.12 0.12 0.00 0.02 0.00 -0.04 
(0.32) (0.33) 
2 0.06 0.06 0.05 0.06 0.04 0.01 -0.01 -0.03 
(0.23) (0.23) 
3 0.13 0.14 0.12 0.13 0.02 0.02 -0.03 0.00 
(0.33) (0.34) 
4 0.18 0.16 0.18 0.18 0.01 -0.04 0.02 0.04 
(0.38) (0.38) 
5 0.08 0.09 0.09 0.09 -0.03 0.00 -0.02 -0.01 
(0.29) (0.29) 
6 0.14 0.15 0.16 0.15 -0.05 -0.01 -0.01 0.04 
(0.37) (0.36) 
7 0.12 0.11 0.12 0.12 -0.02 -0.02 0.05 0.04 
(0.33) (0.32) 
8 0.17 0.16 0.15 0.15 0.04 0.03 0.00 -0.05 
(0.36) (0.36) 
Racial/ethnic composition of the school (fraction of students in the school) 
Black 0.63 0.62 0.65 0.62 -0.04 -0.02 0.01 -0.01 
(0.32) (0.32) 
Hispanic 0.22 0.22 0.21 0.22 0.03 -0.01 0.00 0.01 
(0.25) (0.24) 
White 0.11 0.12 0.10 0.12 0.02 0.03 -0.01 0.00 
(0.19) (0.20) 
Other 0.04 0.04 0.04 0.04 0.04 0.04 -0.01 0.00 
(0.05) (0.05) 
School-level averages across students 
ELA achievement test -0.09 -0.07 -0.10 -0.07 0.01 0.01 0.02 0.04 
score (0.56) (0.57) 
Math achievement -0.07 -0.05 -0.08 -0.06 0.02 0.01 0.02 0.02 
test score (0.54) (0.54) 
Fraction of days 0.89 0.89 0.89 0.89 0.01 0.01 0.09 0.11 
absent (0.11) (0.12) 
Number of 0.12 0.13 0.13 0.13 -0.04 -0.02 -0.07 -0.07 
suspensions (0.15) (0.18) 
Composite of 4.57 4.56 4.57 4.56 -0.01 -0.01 0.23 0.25 
covariates (0.10) (0.11) 


ELA is English language arts. STAR is School Transparency and Reporting Framework. 
Note: This table explores nonresponse bias through the mean, standard deviation, and correlation with the scale across covariates. 
Source: Authors’ analyses based on survey and administrative data provided by the District of Columbia Public Schools, 2017/18 to 2018/19. 
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Table B23. Difference between survey sample and original sample for the composite of covariates with and without nonresponse weights 
Difference in composite of covariates between survey sample and original sample in standard deviation units 


2017/18 2018/19 2019/20 2017/18 & 2018/19 2018/19 & 2019/20 2017/18 & 2019/20 

Without With Without With Without With Without With Without With Without With 
WEE weights weights weights weights weights weights weights weights weights weights weights weights 
(a) Student survey 
Perseverance 0.19 -0.03 0.13 -0.02 0.22 0.00 -0.20 0.07 -0.31 0.10 -0.40 0.04 
Self-management 0.18 -0.03 0.19 -0.03 0.23 -0.04 -0.06 0.05 -0.30 0.06 -0.36 0.04 
Self-efficacy 0.24 -0.02 0.20 -0.02 0.35 -0.03 -0.20 0.09 -0.20 0.09 -0.31 0.03 
Social awareness 0.23 -0.03 0.23 -0.03 0.32 -0.05 -0.18 0.07 -0.40 0.06 -0.39 0.05 
Rigorous expectations 0.25 -0.03 0.22 -0.03 0.36 -0.04 -0.15 0.05 -0.24 0.09 -0.34 0.00 
Student satisfaction 0.22 -0.03 0.19 -0.03 0.32 -0.07 -0.16 0.07 -0.25 0.07 -0.35 0.05 
Sense of belonging 0.22 -0.03 0.19 -0.03 0.32 -0.05 -0.16 0.05 -0.16 0.07 -0.30 0.06 
(b) Teacher survey 
Perseverance 0.01 0.00 0.00 0.01 na na na na na na na na 
Rigorous expectations -0.01 0.00 -0.01 0.00 na na na na na na na na 


na is not applicable. 
Note: This table displays the difference in the composite of covariates between the survey sample and original sample, with and without nonresponse weights, in standard deviation units. 
Source: Authors’ analyses based on survey and administrative data provided by the District of Columbia Public Schools, 2017/18 to 2019/20. 


Research question 1. How do average SEL competencies and school experiences differ across grade levels and 
change for individual students between years? Do student and teacher reports of school experiences and SEL 
competencies change in similar ways across grade levels? To what extent do the average differences in students’ 
SEL competencies and school experiences across grades differ by the type of students (such as students classified 
by gender, race/ethnicity, and academic achievement)? How are individual students’ reports of SEL competencies 
and school experiences associated between years, and how does that association compare to that of other 
variables (such as achievement test scores, absences, and suspensions) ? 


Using nonresponse weights, the study team calculated the average value of students’ self-reported SEL 
competencies and school experiences for each grade and age for the full sample of students. The analyses were 
conducted separately by grade and age to account for the possibility that students might repeat grades so that 
not all students in a given grade will be the same age. Using nonresponse weights, the study team calculated the 
average value of teachers’ reports of SEL competencies and school experiences by grade level of the teachers’ 
school (elementary, middle, or high school). To provide a comparison to the teacher averages, the study team also 
calculated averages of the students’ reports of SEL competencies and school experiences by school level. To assess 
whether the measures varied across grades and school levels, the study team conducted F-tests of the null 
hypothesis that the measures were equal across grades and school levels. The resulting p-value provided evidence 
about whether the measures differed significantly across grades. 


The student analyses were also conducted separately by student characteristics, including gender, cohort, 
race/ethnicity, geographic ward, special education status, English learner status, at-risk status, performance on 
achievement tests, suspension status, and chronic absenteeism status. The subgroups were defined based on the 
time of the reporting for a particular year. 


Other analyses examined year-to-year correlations between SEL competencies and school experiences, as well as 
academic outcomes, shedding light on the degree to which individual students’ reports are consistent year-to- 
year. To do so, the study team calculated pairwise correlations between two measurements of the same SEL 
competency or school experiences in consecutive years, using nonresponse weights. To provide information on 
the direction of the relationship, the correlations were not constrained to be positive. If a correlation is closer to 
O, it suggests that students’ SEL measures change from year to year because the measure in one year is not highly 
correlated with the measure in another year. These statistics were calculated separately by student 
characteristics, including gender, race/ethnicity, geographic ward, special education status, English learner status, 
at-risk status, performance on achievement tests, suspension status, and chronic absenteeism status. 


Research question 2. To what extent do year-to-year changes in individual students’ SEL competencies and 
school experiences differ across schools? 


To examine the variation in measures within versus across schools, the study team calculated the intraclass 
correlation coefficient (ICC), a statistic that represents the extent to which a variable fluctuates within groups 
versus across groups. The study team calculated the ICC for SEL competencies and school experiences, as well as 
changes in SEL competencies and school experiences across years. The results provided a way to decompose the 
overall variance into within- and between-school components. In particular, the ICC is the fraction of a variable’s 
total variance that arises across schools, so the units are comparable across different variables.”° To provide a 
benchmark, ICCs were calculated for other academic measures, including math and ELA achievement, in-seat 
attendance, and number of suspensions. For the analyses of changes in measures, the school assignment was 


20 In particular, the ICC is invariant to linear changes in scale. For example, the ICC would not change if a variable were multiplied by a 
constant. However, the ICC is not invariant to nonlinear changes in scale. For example, the ICC may change if the natural log of the variable 
were used instead of the original variable. 
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based on the second year of the measurement, because that is the school that (typically) students will have 
attended the longest between when the two measures were collected. When calculating the ICCs, nonresponse 
weights were used. 


The ICCs for changes in measures were translated into a measure that represents the expected difference in 
percentile points between students who attend a school with a high average change for each measure compared 
to a school with an average change for each measure. This translation was designed to provide a more intuitive 
metric for interpreting the results than the ICC. A high-change school was defined as being one standard deviation 
above average in terms of change within DCPS. Although this metric was presented as a comparison of student 
growth between two types of schools, it was based on the ICC as calculated using the full sample of schools. 


To make the translation, the study team applied the following formula: 


Dif ference in percentiles = 100 x ® (vice x /2[1- corr(Mr+1,M,)1) — 50, 


where © is the standard normal cumulative distribution function and corr(M,.,,, M;) is the correlation between 
the measure in successive years. 


The formula is based on translating the ICC into standard deviation units. Let of be the variance of change in the 
measure between schools, a# be the total variance in change in the measure, and oj, be the total variance of the 
levels of the measure. 


By construction, ICC = of /a2, which implies that og = VICC x rt X dy. In other words, one standard deviation 
M 


of change in the measure between schools is equivalent to VICC x <r standard deviations of the measure. 
M 


Define a high-change school as one with change that is one school-level standard deviation above the mean and 
a typical school as one with change at the mean. For two students who initially start off with the same level of the 


measure, the student who attends the high-change school is expected to have a measure that is V/CC x = 
M 


standard deviations higher after one year than the student who attends the typical school. Suppose that the 
student in the typical school has an outcome (M;.,) that is equal to the mean of the outcome after one year (,2). 
Assuming that M;,, follows a normal distribution, M;4;~N (wu, oj), then the student’s outcome in the typical 
school is at the 50th percentile. The outcome of the student in the high-change school is then expected to be p + 
VICC X st om. The percentile for the student in the high-change school is therefore given by 100 x 
M 
o( ICC Xx or), where ®() is the standard normal cumulative distribution function. Because = = 
M M 
2[1 — corr(M,,,,M,)], the difference in percentiles between the student in the high-change school and typical 
school can be written as: 


Dif ference in percentiles = 100 x ®(VICC x ./2[1 — corr(Mi41,M;)]) — 50. 


Research question 3. How do measures of SEL competencies and school experiences relate to future outcomes, 
and how do they complement other available data for predicting future outcomes? To what extent do individual 
SEL competencies and school experiences relate to student outcomes measured one and two years later (such as 
achievement test scores, absences, suspensions, and whether a student feels loved, challenged, and prepared)? 
When other data are available—such as demographic information, achievement test scores, absences, and 
suspensions—to predict students’ future outcomes, to what extent does adding measures of SEL competencies and 
school experiences improve the predictive power and accuracy of those predictions? Which types of data and 
statistical models could best help DCPS classify whether students are at risk of having negative outcomes? 
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Predictive power of individual predictors 


To gauge the predictive power of each predictor, the analyses included models that examine the relationship 
between each key outcome and each predictor variable. The predictors included SEL competencies, school 
experiences, and academic measures (achievement test scores, in-seat attendance, and number of suspensions). 
The study team summarized predictive power using two complementary statistics: pairwise correlations and 
classification accuracy. 


1. Pairwise correlations. The study team used pairwise correlations to summarize the strength of a predictor 
and an outcome. All predictor variables were continuous. The correlations were calculated using nonresponse 
weights. 


2. Classification accuracy of individual predictors. To provide a measure of how well individual predictors could 
identify students at risk of poor outcomes, the study team estimated the predictor’s classification accuracy. 
For binary outcomes, predictor variables can be used to classify whether a student is likely to achieve an 
outcome or not—for example, whether they are likely to graduate from high school or not. A good predictor 
would have a higher classification accuracy in that it correctly classified a higher fraction of outcomes. 


For outcome variables with little variation, total classification accuracy can be misleading. For example, consider 
an outcome that 95 percent of students achieve with success. Without additional information, classifying all 
students as successful would result in a relatively high classification accuracy of 95 percent. The baseline accuracy 
(accuracy of the null model) is the percentage of students who would be correctly classified if all students were 
classified with the most prevalent value for each outcome (in this example, 95 percent). Therefore, a predictor 
that classified 95 percent of students as successful would not add much value. For this reason, the study team 
examined improvements in classification accuracy relative to baseline accuracy. For example, if 95 percent of 
students achieved a positive outcome and the predictor yielded a 97 percent classification accuracy, the 
improvement in classification accuracy would be 2 percentage points. 


The study team calculated classification accuracy in five steps: 


1. The study team converted any continuous outcomes into meaningful binary variables. For example, in-seat 
attendance was converted into whether a student was chronically absent or not. 

2. The study team estimated probit models of the outcome as a function of the predictor. This model provided 
an estimate of the probability that each student would have a given outcome (for example, the probability 
that a student would be chronically absent). The model was estimated using nonresponse weights. 

3. Based on the estimated probabilities, the study team determined a cutoff in terms of the probabilities that 
would maximize the classification accuracy, such that students with an estimated probability above or below 
the cutoff would be classified as likely to have that outcome or not. Using nonresponse weights, the 
classification accuracy was calculated for potential cutoffs in increments of 0.01. The study team selected the 
cutoff that yielded the highest classification accuracy out of those that were tested. The classification accuracy 
associated with this cutoff was reported as the total classification accuracy (ACC). 

4. Using nonresponse weights, the study team calculated the percentage of students who had a value of 1 for 
the outcome (P). 

5. The improvement in accuracy ([ACC) was calculated by subtracting from the classification accuracy the 
baseline accuracy of classifying all students the same way given information on P: [ACC = ACC — 
max (P,1— P). 


REL 2021-114 B-30 


Predictive power of groups of variables 


The study also considered the predictive power of groups of variables by estimating multivariate models that 
examine the relationship between students’ outcomes and groups of the predictor variables. The analyses 
examined the predictive power of all SEL competencies and school experiences combined, all academic measures 
combined, and all demographic characteristics combined. The analyses also examined the predictive power of 
combinations of the groups of variables, which provides information on the incremental predictive power—the 
extent to which one group of variables adds to the predictive power of another group of variables. To assess the 
predictive power of groups of variables, the study team adopted three approaches: 


1. 


Multivariate correlation. A multivariate correlation is a measure of strength between a single variable of 
interest and a group of other variables. It ranges from 0 to 1. The higher the multivariate correlation, the more 
related the variable is to the group of other variables. The study team calculated the multivariate correlation 
as the square root of the adjusted R-squared from an ordinary least squares regression of each outcome ona 
group of predictors. The regressions accounted for nonresponse weights. The incremental predictive power 
is the difference in multivariate correlations between different groups of predictors. For example, the 
incremental predictive power of the SEL competencies and school experiences is the difference between the 
multivariate correlation based on a regression that uses all the predictors and a multivariate correlation based 
ona regression that uses all the predictors except the SEL competencies and school experiences. The predictor 
variables included a combination of continuous and categorical variables. For categorical variables, the models 
included indicators for each category, excluding a reference category. 


Classification accuracy of groups of variables based on probit models. Using the methodology described in 
the analyses of individual predictors, the study team estimated the percentage of students that each group of 
predictors could correctly classify. Each predictor was included separately in the model. In other words, the 
models did not include interaction terms between the predictors. To assess the incremental predictive power, 
the accuracies were compared across groups of predictors. For example, the incremental predictive power of 
the SEL competencies and school experiences is the difference between the percentage that are correctly 
classified from the model that uses all the predictors and the percentage that are correctly classified from the 
model that uses only the predictors based on variables other than the SEL competencies and school 
experiences. This metric provides a practical sense of how much the inclusion of SEL competencies and school 
experiences could boost the ability to predict student outcomes. 


Classification accuracy of groups of variables based on random forest models. Random forests are a machine 
learning method commonly used for prediction problems. The study team implemented random forest 
models using the same groups of variables as in the probit approach, including SEL competencies, school 
experiences, academic measures, and basic demographics. The models were estimated with the ranger 
package for the statistical programming language R (Wright & Ziegler, 2017). The study team used standard 
procedures for selecting the options for the analysis. Specifically, the study team constructed 500 decision 
trees for each model. For each decision tree, the study team sampled data with replacement and evaluated 
the square root of the number of available predictors at each split. The modeling approach was designed to 
align with the probit methodology (for example, over-sampling observations with larger nonresponse weights) 
in order to compare the incremental predictive power afforded by different sets of predictors.7? 


21 The random forest predictions are based on out-of-bag samples—the predictions of the decision trees in which a given case was not 


included in the bootstrapped sample. As a sensitivity, the team also estimated the random forest models using 10-fold cross-validation. 
These cross-validated models produced results which closely aligned with the relative accuracy rates of the benchmark approach, always 


falling within + 3 percentage points. 
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Linking “credits behind in grade 9” to high school graduation 


The study team used a similar type of analysis to identify the extent to which grade 8 students are likely to fall 
behind academically, focusing on the middle and high school sample. This analysis provides context for a key 
subgroup analysis that examined how grade 8 SEL competencies and school experiences relate to credits earned 
in high school. To understand which SEL competencies and school experiences might be most important for 
transitioning from grade 8 to high school, the study team estimated pairwise correlations between the SEL 
competencies and school experiences of grade 8 students in 2017/18 and the extent to which students were 
behind in terms of credits earned in 2018/19. DCPS currently predicts whether students are falling behind using a 
measure based on the number of credits by which a student is behind for his or her grade level (credits behind). 


Ideally, the study would have examined the probability of graduating from high school given data from grade 8, 
but the study only had access to three years of data, so it is not possible to calculate this probability directly. 
Instead, using data on credits behind from the 2017/18 and 2018/19 school years and graduation data from the 
2018/19 school year, the study examined the extent to which credits behind in grade 9 reflected the likelihood of 
graduating by linking data from several cohorts. This analysis provides a sense of whether the grade-8 SEL 
competencies and school experiences are related to high school graduation through their link to the number of 
credits behind in grade 9. For example, if grade-8 SEL competencies relate to grade-9 credits behind and grade-9 
credits behind relate to high school graduation, then grade-8 SEL competencies could be an indicator for likely 
high school graduation. One limitation of this approach is that it assumes that the models are the same across 
subsequent cohorts of students. The analyses proceeded in two steps: 


Step 1: Establishing the relationship between credits behind in grade 9 and the probability of graduating high 
school. Using ordinary least squares regressions, the study team estimated the relationship between credits 
behind in each grade as a function of credits behind in the previous grade. In addition, using a probit model, the 
study team estimated the probability of graduating given the number of credits behind in grade 11. This 
information was combined to form a probability of graduating given the grade-9 credits behind. Formally, the 
following equations were estimated using data from several separate cohorts of students: 


Diy = G11 + P11Mia + Vina 
(Equation B1) Mi11 = &10 + P1oMir0 + Vito 
Mito = 209 + PooMiog + Vioo, 


where Mi; is the number of credits behind at time t, D;,, is the propensity to graduate from high school, and v;11, 
Viio, and Vjgq are error terms assumed to be independent from each other. Each equation was estimated 
separately for a single cohort using data from the 2017/18 and 2018/19 school years. For example, P99 was 
estimated using the cohort of students who were in grade 9 during the 2017/18 school year. Estimates of a, and 
Pt were recursively substituted to relate D;,. to Mio. 


Step 2: Estimating the extent to which credits behind in grade 9 can accurately classify high school graduation. 
The study team conducted a simulation to determine the predictive accuracy of grade-9 credits behind in 
classifying high school graduation. Because the sample did not include students in grade 9 who could have 
graduated in the timeframe of the study, it was not possible to directly calculate the predictive accuracy of grade- 
9 credits behind. Instead, the study team conducted a simulation using the following steps: 


1. The data were subset to students in grade 9 in 2017/18. 

2. Using the estimated residual variance of Vig9 (G29), a Simulated error term was drawn for each student from 
the normal distribution: N (0, 6259). 

3. Based on estimates of @gq and Pog and the simulated error, a simulated value of credits behind in grade 10 
was formed for each student (M19). 
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4. Using Miso; step (1) was repeated step to form simulated values of grade-11 credits earned and high school 
graduation. 

5. Then, treating the simulated data as real, the estimated values of equation B1 were used to classify students 
as likely to graduate or not based on their grade-9 credits behind. This step provided an estimate of the 
accuracy of grade-9 credits behind in classifying high school graduation. 

6. Steps (1) through (5) were repeated 500 times to form 500 estimates of the classification accuracy. The 
average across those estimates was used to form an overall classification accuracy. 


Research question 4. How do measures of perseverance and rigorous expectations align across students, 
parents, and teachers? Across schools, to what extent do survey reports on these measures from students, parents, 
and teachers align? Is alignment associated with characteristics of schools (such as the demographic 
characteristics of their student population and the schools’ accountability ratings) and response rates on the 
survey? 


Because of data limitations, the analyses focused on estimating correlations among school-level averages of SEL 
competencies and school experiences among respondent types. Ideally, the analysis would have examined the 
alignment between students, parents, and teachers at the student level, but this analysis is not possible for two 
reasons: teacher reports about the SEL competencies of groups of students (not individual students) and parent 
responses are not linked to their child.”? Instead, the analyses examined alignment at the school level by forming 
averages of the reports from each type of respondent for each school. The study team then calculated pairwise 
correlations among the averages. 


To examine overall alignment, the study team estimated the average student, parent, and teacher reports for 
each school. The correlations among the measures suggest whether the responses by respondent type tend to 
move together, but these correlations do not inform whether the overall responses align in terms of their levels. 
To do so, the analyses explored two types of measures: 


1. The difference between each pair of respondent types. These values could be negative or positive, so they 
indicate which respondent type had the higher value. 

2. The absolute difference between pairs of respondent types. The absolute differences are constrained to be 
positive so they indicate the magnitude of the differences but do not indicate which respondent types had 
the higher value. 


The analyses examined the extent to which these measures of alignment vary across schools and whether 
alignment relates to other characteristics of schools, including student composition (percentage of students by 
race/ethnicity, English learner status, and special education status), STAR rating, geographic ward, and response 
rates on the surveys. For continuous school characteristics (for example, percentage of students in various 
demographic groups), the study team calculated correlations between the measures of misalignment and the 
school characteristic. For categorical school characteristics (for example, geographic ward), the study team 
calculated the levels of misalignment separately for each category. 


The study team did not adjust for nonresponse in these analyses, because it was not possible to explore 
nonresponse bias for parents. Because DCPS does not have information on the number of parents eligible for the 
survey, it is not possible to calculate response rates on the parent survey. For consistency, the study team did not 


22 The items in the teacher survey ask teachers to rate the competencies of the students that they teach. Therefore, teachers in different 
types of schools will rate different groups of students. For example, elementary school teachers might rate a group of students they teach 
for all school subjects, whereas high school teachers might rate the competencies of students within a particular subject. 
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use student or teacher nonresponse weights for results reported in the main text. As a sensitivity check, the study 
team conducted analyses using student and teacher nonresponse weights to confirm the findings generalize to 
the population of respondents at each school. 


Benchmarks 


To describe the strength of the correlations in this context, this study adopted benchmarks based on past 
evidence on the correlations between cognitive tests (IQ tests and achievement tests) and other future 
academic outcomes. Historically, cognitive tests have been most widely used in educational assessment 
(Heckman & Kautz, 2014), so their predictive power serves as a natural comparison for other types of 
assessments that are starting to be used more frequently, including measures of SEL competencies and school 
experiences. 


This study described ranges of correlations as follows: 0.0 to 0.09 is low, 0.10 to 0.19 is moderate, 0.20 to 0.29 is 
substantive, and 0.30 and above is high. The study team viewed correlations above 0.30 as high, because this 
value represents the upper end of the estimated correlations between cognitive tests and other future academic 
outcomes from prior research (table B24). Given that many estimates of the correlations between cognitive 
tests and other academic outcomes fell in the range of 0.20 to 0.29, the study viewed this range as substantive. 
Correlations in the range of 0.10 to 0.19 were classified as moderate because they were approximately half the 
magnitude of those classified as high and, thus, may still be meaningful. 


Table B24. Correlations between cognitive tests and other future academic outcomes in existing literature 


Type of test Outcome Correlation Source 
Achievement test High school graduation 0.33 Kautz & Zanoni (2020) 
Achievement test College grades 0.15-0.28° Noftle & Robins (2007) 
Achievement test Graduated college 0.27 Galla et al. (2019) 
Achievement test Completed a bachelor’s degree 0.43 Heckman & Kautz (2012) 
1Q test Completed a bachelor’s degree 0.35 Heckman & Kautz (2012) 
1Q test Academic performance 0.23 Poropat (2009) 

1Q test Years of educational attainment 0.07-0.20° Almlund et al. (2011) 


* Based on standardized regression coefficients, which have a similar interpretation to a correlation. 


To describe the magnitude of differences in SEL competencies and school experiences between groups of 
students, this study adopted benchmarks based on past evidence on the effect of school-based SEL programs. 
This evidence provides a natural benchmark in this context because it suggests the extent to which SEL 
programming could reduce differences between groups of students. Therefore, differences that exceed the 
effects of school-based SEL programs could be viewed as large because the differences might be challenging to 
eliminate through SEL programming. 


To set the benchmarks, the study team drew on a meta-analysis of school-based SEL programs. The meta- 
analysis estimated effects in standard deviation units of 0.57 on SEL skills, 0.23 on attitudes, 0.24 on positive 
social behavior, 0.22 on conduct problems, and 0.24 on emotional distress (Durlak et al., 2011). Because these 
categories of outcomes span the SEL competencies and school experience variables in this study, the average 
effect —0.30 standard deviations—provided a relevant summary of the overall effect. The study team, therefore, 
described differences of 0.30 standard deviations and above as large. For consistency with the benchmarks for 
correlations, the study described smaller differences between groups as follows: 0.0 to 0.09 standard deviations 
is small, 0.10 to 0.19 standard deviations is moderate, and 0.20 to 0.29 is substantive. 
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Limitations 

Four main limitations should be kept in mind. First, as discussed in the main text, the study examined correlational 
relationships between variables, not causal ones. Therefore, the findings do not indicate that improvements in 
SEL competencies or school experiences would necessarily improve students’ outcomes. 


Second, the study focused on two years of data on each students’ SEL competencies and school experiences, 
which affects the interpretation of the findings. The analyses of how SEL competencies and school experiences 
evolve with grade relied on comparing different students in different cohorts and grades rather than examining 
changes in the same student over time. This approach requires that SEL competencies do not evolve in different 
ways across cohorts of students. Supporting this requirement, the trends were similar across the two cohorts in 
the analysis sample. As discussed in the main text, the study cannot rule out that the U-shaped pattern emerged 
because the composition of students changed across grades. Although future research is needed to determine 
the source or sources of this pattern, the results are still useful for DCPS, because the estimates apply to the 
students who are enrolled and who DCPS can still reach through programming or policies. For example, DCPS can 
use these estimates to help inform decisions about when to target supports for enrolled students. 


Third, data limitations reduced the generalizability of the study in several ways. The study focused on one school 
district during a relatively short timeframe and on one of many possible surveys that school districts could use, 
potentially limiting the generalizability. Because the surveys were administered recently, the study could not 
examine longer-term trends and outcomes, and the measure of alignment across respondent types could reflect 
differences in survey response rates (which cannot be adjusted for, given a lack of data on parents). 


Fourth, the COVID-19 pandemic affected both the administrative and survey data from the 2019/20 school year, 
which limited its use in this study. The administration of the survey, which is typically administered online to 
students in school in early March, was only available online for students to complete at home once students could 
no longer attend school in-person—resulting in a lower response rate for students. For all respondents, the survey 
was not likely a top priority given the pandemic’s disruption of normal life, and those who did respond might have 
responded differently as a result. For these reasons, the main analyses did not include the 2019/20 survey data. 
In addition, certain annual standardized tests were canceled in spring 2020, including the PARCC exams and spring 
administrations of the SAT and PSAT. 
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Appendix C. Supporting analysis 
This appendix presents supporting analyses for each research question. 


Research question 1. How do average social and emotional learning (SEL) competencies and school 
experiences differ across grade levels and change for individual students between years? Do student 
and teacher reports of SEL competencies and school experiences change in similar ways across grade 
levels? To what extent do the average differences in students’ SEL competencies and school experiences 
across grades differ by the type of students (such as students classified by gender, race/ethnicity, and 
academic achievement)? How are individual students’ reports of SEL competencies and school 
experiences associated between years, and how does that association compare to that of other 
variables (such as achievement test scores, absences, and suspensions)? 


Students’ average SEL competencies and school experiences exhibited similar patterns across ages as they did 
across grades (figure C1). As in the analyses across grades, the averages across ages for both SEL competencies 
and school experiences exhibited a U-shaped pattern. Some measures varied more than others. Self-management 
was relatively stable across age, whereas sense of belonging varied the most. 


Figure C1. Students self-reported SEL competencies and school experiences exhibited similar patterns by age 
as by grade 


(a) SEL competencies (b) School experiences 
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SEL is social and emotional learning. 

Note: The figure shows for each student ages 8-18 the mean of each SEL competency and school climate scale (described in appendix B). The means were 
calculated using nonresponse weights, as described in appendix B. 

Source: Authors’ analyses based on survey and administrative data provided by the District of Columbia Public Schools, 2017/18 to 2018/19. 


Students’ SEL competencies and school experiences also showed similar patterns across subgroups of students as 
they do in the full sample (tables C1—C7). Across subgroups, students tended to have the lowest SEL competencies 
and school experiences during middle school (grades 6-8) and early high school (grades 9 and 10). However, as 
discussed in the main text, some subgroups tended to have higher levels of SEL competencies and school 
experiences for all grades. 


[ee 
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Table C1. Average level of perseverance by subgroup and grade 


p-value 
from test 
of equality 


across 
grades 
All 3.95 3.93 3.87 3.77 3.63 3.61 3.56 3.59 3.65 3.79 <0.001 


Gender 


Male 3.89 3.86 3.83 3.76 3.63 3.61 3.52 3.59 3.65 3.77 <0.001 
Female 4.02 3.99 3.92 3.77 3.64 3.60 3.60 3.58 3.65 3.80 <0.001 
Absolute value of Oar OA Ole Taha 0.01 0.01 0.10 0.01 0.00 0.04 
the standardized 
difference in 
maximum and 
minimum value 


Survey year 
2017/18 3.95 3.91 3.88 3.78 3.65 3.63 3.50 3.56 3.63 3.80 <0.001 
2018/19 3.96 3.94 3.87 3.75 3.62 3.59 3.62 3.62 3.67 3.77 <0.001 


Absolute value of 0.01 0.04 0.01 0.04 0.04 0.05 0.16** 0.08 0.05 0.04 
the standardized 

difference in 

maximum and 

minimum value 

Race/ethnicity 


Black 4.02 3.99 3.92 3.83 3.69 3.68 3.58 3.63 3.70 3.83 <0.001 
Hispanic 4.00 3.96 3.88 3.73 3.60 3.54 3.52 3.49 3.57 3.72 <0.001 
White 3.70 3.66 3.64 3.54 3.41 3.38 3.44 3.52 3.48 3.50 <0.001 
Other 3.79 3.82 3.74 3.67 3.61 3.50 3.49 3.57 3.64 3.73 <0.001 

Absolute value of 2 : 0.18*** [Opes 

the standardized 

difference in 


maximum and 

minimum value 

At-risk status 
At risk 4.03 4.02 3.93 3.83 3.71 3.66 3.58 3.65 3.69 3.81 <0.001 
Not at risk 3.89 3.85 3.82 3.71 3.58 3.58 3.53 3.52 3.62 3.77 <0.001 

Absolute value of 0.18*** (Ope O.14*** 0.16*** O.17*** 0.10** 0.07 0.17*** 0.09 0.05 

the standardized 

difference in 

maximum and 

minimum value 


Ward 
1 4.00 3.86 3.90 3.70 3.64 3.58 3.44 3.56 3.56 3.78 <0.001 
2 3.95 3.79 3.74 3.69 3.60 3.65 3.66 3.57 3.63 3.59 <0.001 
3 3.68 3.69 3.64 3.58 3.41 3.39 3.44 3.48 3.51 3.59 <0.001 
4 4.04 3.99 3.82 3.83 3.74 3.67 3.64 3.64 3.70 3.75 <0.001 
5 4.09 4.03 3.92 3.87 3.75 3.73 3.63 3.62 3.70 3.86 <0.001 
6 3.89 3.89 3.91 3.84 3.70 3.65 3.59 3.51 3.61 3.78 <0.001 
7 4.07 4.10 3.95 3.83 3.77 3.75 3.66 3.61 3.73 3.87 <0.001 
8 4.02 3.99 4.01 3.83 3.67 3.69 3.54 3.71 3.71 3.75 <0.001 
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Table C1. Average level of perseverance by subgroup and grade (continued) 


p-value 
from test 


of equality 
across 
grades 


Absolute value of 
the standardized 
difference in 
maximum and 
minimum value 
Special education status 


Special education 3.85 3.84 3.79 3.76 3.61 3.57 3.48 3.59 3.49 3.84 <0.001 
Not special 3.97 3.94 3.89 3.77 3.64 3.62 3.58 3.58 3.68 3.77 <0.001 
education 


Absolute value of = 0.16*** 0.13*** 0.13*** 0.01 0.04 0.07 0.13 0.01 0.25** 0.09 
the standardized 

difference in 

maximum and 

minimum value 

English learner status 


English learner 4.00 3.99 3.85 3.74 3.73 3.60 3.56 3.53 3.57 3.74 <0.001 
Not English 3.94 3.92 3.87 3.77 3.62 3.61 3.56 3.59 3.66 3.79 <0.001 
learner 


Absolute value of  0.08* 0.09** 0.03 0.04 0.14* 0.01 0.00 0.08 0.12 0.07 
the standardized 

difference in 

maximum and 

minimum value 

Proficient/college ready in math and ELA 


Proficient/college 3.88 3.83 3.83 3.66 3.56 3.56 3.63 3.68 3.61 3.67 <0.001 
ready ee 
Not proficient/ 3.99 3.97 3.89 3.79 3.66 3.64 3.59 3.58 3.66 3.80 <0.001 


college ready 
Absolute value of = 0.14*** 0.18*** 0.08** 0.17*** 0.13*** 0.10** 0.05 0.13 0.07 OMe ae 
the standardized 
difference in 
maximum and 
minimum value 
Chronic absentee status 


Chronically 3.98 3.98 3.88 3.74 3.61 3.56 3.49 3.56 3.62 3.79 <0.001 
absent 
Not chronically 3.95 3.92 3.87 3.77 3.64 3.63 3.65 3.63 3.71 3.77 <0.001 
absent 


Absolute value of 0.04 0.08* 0.01 0.04 0.04 0.09* 0.21*** 0.09* 0.12** 0.03 
the standardized 

difference in 

maximum and 

minimum value 
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Table C1. Average level of perseverance by subgroup and grade (continued) 


p-value 
from test 
of equality 
EYo vol) 
grades 
Suspension status 
Ever suspended 3.85 3.86 3.75 3.71 3.60 3.54 3.42 3.53 3.58 3.82 <0.001 
Never suspended 3.96 3.93 3.88 3.78 3.64 3.63 3.59 3.60 3.66 3.78 <0.001 


Absolute value of 0.14 0.09 0.17** 0.09 0.05 0.12** 0.22* 0.09 0.10 0.05 
the standardized 

difference in 

maximum and 

minimum value 


Absolute value of the 
standardized difference Small Moderate Substantive 
0.00-0.09 0.10-0.19 0.20-0.29 0.30+ 


ELA is English language arts. 

* Significant at p < .05; ** significant at p < .01; *** significant at p < .001. 

Note: The table shows the average level of perseverance (described in appendix B) for each grade level, over both the full sample and for ten sets of 
subgroups. The means were calculated using nonresponse weights, as described in appendix B. 

Source: Authors’ analyses based on survey and administrative data provided by the District of Columbia Public Schools, 2017/18 to 2018/19. 


Table C2. Average level of rigorous expectations by subgroup and grade 


All 
Gender 

Male 

Female 
Absolute value of 
the standardized 
difference in 
maximum and 
minimum value 
Survey year 

2017/18 

2018/19 
Absolute value of 
the standardized 
difference in 
maximum and 
minimum value 
Race/ethnicity 

Black 

Hispanic 

White 

Other 
Absolute value of 
the standardized 
difference in 
maximum and 
minimum value 
At-risk status 

At risk 

Not at risk 
Absolute value of 
the standardized 
difference in 
maximum and 
minimum value 
Ward 


COYN| DD) MW] Bl] WN rR 
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4.37 


4.31 
4.44 
Once 


4.37 
4.38 
0.01 


4.41 
4.36 
4.29 
4.24 
(0) 2 


4.42 
4.34 
0.10*** 


4.36 
4.28 
4.25 
4.41 
4.41 
4.34 
4.49 
4.41 


4.39 


4.34 
4.44 
On eee 


4.36 
4.42 
0.08*** 


4.41 
4.40 
4.32 
4.28 
0.17*** 


4.44 
4.35 
OMe 


4.30 
4.22 
4.26 
4.49 
4.46 
4.36 
4.50 
4.40 


4.37 


4.30 
4.44 
Onset 


4.34 
4.39 
0.06** 


4.39 
4.36 
4.27 
4.30 
() alSy 


4.40 
4.34 
0.08*** 


4.34 
4.23 
4.27 
4.33 
4.37 
4.45 
4.39 
4.44 


4.06 


4.03 
4.09 
0.08** 


4.04 
4.07 
0.04 


4.07 
4.03 
4.01 
4.11 
0.13 


4.08 
4.04 
0.05* 


3.99 
4.00 
4.04 
4.04 
4.04 
4.08 
4.13 
4.10 


3.88 
3.91 
0.04 


3.88 
3.92 
3.92 
4.00 
0.15* 


3.92 
3.88 
0.05 


3.98 
3.84 
3.90 
3.95 
3.92 
3.86 
3.88 
3.87 


3.82 
3.90 
0.10** 


3.87 
3.86 
3.82 
3.89 
0.09 


3.89 
3.85 
0.05 


3.91 
3.98 
3.84 
3.87 
3.85 
3.81 
3.90 
3.84 


3.65 
3.79 
0.18** 


3.71 
3.73 
3.74 
3.77 
0.08 


3.70 
3.74 
0.05 


3.69 
3.94 
3.66 
3.81 
3.69 
3.78 
3.64 
3.64 


3.69 


3.66 
3.72 
0.08 


3.63 
3.74 
0.14* 


3.68 
3.70 
3.68 
3.75 
0.09 


3.70 
3.67 
0.04 


3.76 
3.88 
3.52 
3.84 
3.53 
3.53 
3.67 
3.66 


3.76 


3.74 
3.79 
0.06 


3.72 
3.80 
0.10* 


3.77 
3.74 
3.72 
3.82 
0.13 


3.77 
3.75 
0.03 


3.76 
3.78 
3.59 
3.83 
3.63 
3.80 
3.77 
3.77 


3.84 


3.83 
3.86 
0.04 


3.81 
3.88 
0.09 


3.87 
3.84 
3.61 
3.79 


3.85 
3.84 
0.01 


3.88 
3.79 
3.55 
3.92 
3.85 
3.95 
3.72 
3.78 


p-value 
from test 
of equality 


across 
grades 
<0.001 


<0.001 
<0.001 


<0.001 
<0.001 


<0.001 
<0.001 
<0.001 
<0.001 


<0.001 
<0.001 


<0.001 
<0.001 
<0.001 
<0.001 
<0.001 
<0.001 
<0.001 
<0.001 
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Table C2. Average level of rigorous expectations by subgroup and grade (continued) 


p-value 
from test 


of equality 
across 
grades 


Absolute value of 
the standardized 
difference in 
maximum and 
minimum value 
Special education status 


Special education 4.26 4.28 4.27 3.96 3.85 3.75 3.62 3.64 3.63 3.86 <0.001 
Not special 4.39 4.41 4.39 4.08 3.91 3.88 3.75 3.70 3.79 3.84 <0.001 
education 


Absolute value of 0.17*** 0.17*** 0.15*** 0.15** 0.08 0.17** 0.17 0.08 0.20* 0.03 
the standardized 

difference in 

maximum and 

minimum value 

English learner status 


English learner 4.34 4.36 4.27 3.86 3.89 3.83 3.71 3.75 3.76 3.93 <0.001 
Not English 4.38 4.39 4.38 4.08 3.90 3.86 3.72 3.68 3.76 3.84 <0.001 
learner 


Absolute value of 0.05 0.04 0.14*** 0.28*** 0.01 0.04 0.01 0.09 0.00 0.12 
the standardized 

difference in 

maximum and 

minimum value 

Proficient/college ready in math and ELA 


Proficient/college 4.40 4.42 4.41 4.13 4.00 3.94 3.92 3.92 3.82 3.85 <0.001 
ready 
Not proficient/ 4.37 4.38 4.36 4.04 3.86 3.85 3.71 3.68 3.74 3.83 <0.001 


college ready 
Absolute value of 
the standardized 
difference in 
maximum and 
minimum value 
Chronic absentee status 


0.04* 0.06*** 0.18*** 


Chronically 4.38 4.41 4.30 3.98 3.79 3.75 3.63 3.63 3.72 3.83 <0.001 
absent 
Not chronically 4.37 4.39 4.38 4.08 3.94 3.91 3.85 3.79 3.86 3.88 <0.001 
absent 


Absolute value of 0.01 0.03 0.10** 0.13** 0.19*** 0.20*** 0.28*** 0.20*** 0.18*** 0.06 
the standardized 

difference in 

maximum and 

minimum value 
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Table C2. Average level of rigorous expectations by subgroup and grade (continued) 


p-value 
from test 


of equality 
across 
grades 


Suspension status 
Ever suspended 4.36 4.30 4.31 3.93 3.74 3.71 3.48 3.62 3.68 3.78 <0.001 
Never suspended 4.37 4.39 4.37 4.08 3.93 3.90 3.78 3.70 3.77 3.85 <0.001 

0.01 0.19*** [RAR Sane Ai 0.10 0.12 


Absolute value of 
the standardized 
difference in 
maximum and 
minimum value 


Absolute value of the 


standardized difference Small Moderate Substantive esse 


0.00—-0.09 0.10—0.19 0.20-0.29 0.30+ 


ELA is English language arts. 

* Significant at p < .05; ** significant at p < .01; *** significant at p < .001. 

Note: The table shows the average level of rigorous expectations (described in appendix B) for each grade level, over both the full sample and for ten sets 
of subgroups. The means were calculated using nonresponse weights, as described in appendix B. 

Source: Authors’ analyses based on survey and administrative data provided by the District of Columbia Public Schools, 2017/18 to 2018/19. 


Table C3. Average level of self-efficacy by subgroup and grade 


p-value 
from test 
of equality 
across 
grades 
All 3.92 3.92 3.87 3.72 3.53 3.50 3.40 3.40 3.45 3.61 <0.001 
Gender 
Male 3.89 3.91 3.86 3.76 3.59 3.55 3.42 3.46 3.52 3.64 <0.001 
Female 3.95 3.93 3.89 3.67 3.47 3.44 3.38 3.34 3.37 3.58 <0.001 
Absolute value of 0.07** 0.02 0.04 0.11*** 0.14*** 0.13*** 0.05 0.14** 0.18*** 0.07 
the standardized 
difference in 
maximum and 
minimum value 


Survey year 
2017/18 3.91 3.92 3.88 3.72 3.55 3.51 3.37 3.34 3.42 3.60 <0.001 
2018/19 3.93 3.93 3.86 3.72 3.51 3.49 3.43 3.47 3.48 3.61 <0.001 
Absolute value of 0.02 0.01 0.02 0.00 0.05 0.02 0.07 0.16** 0.07 0.01 


the standardized 
difference in 
maximum and 
minimum value 
Race/ethnicity 


Black 3.95 3.93 3.88 3.73 3.53 3.50 3.39 3.43 3.48 3.66 <0.001 
Hispanic 3.92 3.90 3.80 3.62 3.46 3.39 3.38 3.31 3.36 3.47 <0.001 
White 3.83 3.96 3.97 3.81 3.61 3.64 3.52 3.46 3.46 3.48 <0.001 
Other 3.80 3.87 3.79 3.74 3.55 3.53 3.41 3.37 3.34 3.50 <0.001 


Absolute value of 0.18*** 0.11 Ope aamOwomea 0.18*** 

the standardized 

difference in 

maximum and 

minimum value 

At-risk status 
At risk 3.97 3.96 3.86 3.72 3.51 3.47 3.39 3.44 3.50 3.61 <0.001 
Not at risk 3.88 3.90 3.88 3.72 3.54 3.52 3.41 3.36 3.40 3.61 <0.001 

Absolute value of 0.11***  0.07** 0.02 0.00 0.04 0.06 0.02 0.10* 0.12* 0.00 

the standardized 

difference in 

maximum and 

minimum value 


0.17** 0.18** 0.17* BO@memae 


Ward 
1 3.91 3.77 3.86 3.58 3.55 3.50 3.27 3.37 3.38 3.56 <0.001 
2 3.85 3.83 3.78 3.70 3.52 3.55 3.49 3.40 3.37 3.45 <0.001 
3 3.75 3.89 3.82 3.69 3.49 3.47 3.42 3.29 3.28 3.36 <0.001 
4 3.97 3.97 3.85 3.72 3.50 3.52 3.58 3.41 3.57 3.64 <0.001 
5 3.94 3.92 3.79 3.77 3.59 3.55 3.37 3.38 3.50 3.66 <0.001 
6 3.90 3.93 3.91 3.78 3.57 3.50 3.32 3.28 3.34 3.59 <0.001 
7 4.09 4.02 3.92 3.72 3.55 3.55 3.43 3.47 3.51 3.57 <0.001 
8 3.92 3.93 3.92 3.72 3.51 3.45 3.41 3.57 3.50 3.65 <0.001 
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Table C3. Average level of self-efficacy by subgroup and grade (continued) 


p-value 
from test 


of equality 
across 
grades 


Absolute value of OD Ages 
the standardized 
difference in 
maximum and 
minimum value 


Special education status 


Special education 3.76 3.77 3.71 3.67 3.43 3.43 3.32 3.36 3.32 3.63 <0.001 
Not special 3.95 3.95 3.91 3.73 3.55 3.51 3.42 3.41 3.48 3.60 <0.001 
education 


Absolute value of 0.23*** 0.22*** 0.24*** 0.07 0.14** 0.10 0.12 0.06 0.19* 0.04 
the standardized 

difference in 

maximum and 

minimum value 

English learner status 


English learner 3.91 3.88 3.72 3.55 3.53 3.41 3.39 3.42 3.42 3.54 <0.001 
Not English 3.92 3.93 3.89 3.73 3.53 3.51 3.40 3.40 3.45 3.61 <0.001 
learner 


Absolute value of 0.01 0.06 0.20*** 0.22*** 0.00 0.12* 0.01 0.02 0.04 0.08 
the standardized 

difference in 

maximum and 

minimum value 

Proficient/college ready in math and ELA 


Proficient/college 3.98 4.06 4.07 3.86 3.72 3.74 3.60 3.42 3.50 3.55 <0.001 
ready 
Not proficient/ 3.89 3.86 3.81 3.67 3.46 3.44 3.37 3.35 3.41 3.58 <0.001 


college ready 
Absolute value of 
the standardized 
difference in 
maximum and 
minimum value 


0.11*** (ames 


Chronic absentee status 


Chronically 3.91 3.89 3.77 3.65 3.42 3.38 3.33 3.39 3.43 3.62 <0.001 
absent 
Not chronically 3.92 3.93 3.89 3.74 3.56 3.55 3.50 3.43 3.49 3.56 <0.001 
absent 


Absolute value of 0.01 0.05 OMA Sends 0.17*** OBRORR sO 0ge se 0.05 0.07 0.07 
the standardized 

difference in 

maximum and 

minimum value 
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Table C3. Average level of self-efficacy by subgroup and grade (continued) 


p-value 
from test 


of equality 
across 
grades 


Suspension status 
Ever suspended 3.83 3.75 3.75 3.65 3.49 3.45 3.30 3.45 3.41 3.68 <0.001 
Never suspended 3.92 3.93 3.88 3.73 3.53 3.51 3.42 3.39 3.45 3.60 <0.001 


Absolute value of 0.11 O22 0.16** 0.10 0.05 0.07 0.14 0.07 0.05 0.10 
the standardized 

difference in 

maximum and 

minimum value 


Absolute value of the 
standardized difference Small Moderate Substantive 


0.00—0.09 0.10—-0.19 0.20-0.29 0.30+ 


ELA is English language arts. 

* Significant at p < .05; ** significant at p < .01; *** significant at p < .001. 

Note: The table shows the average level of self-efficacy scale (described in appendix B) for each grade level, over both the full sample and for ten sets of 
subgroups. The means were calculated using nonresponse weights, as described in appendix B. 

Source: Authors’ analyses based on survey and administrative data provided by the District of Columbia Public Schools, 2017/18 to 2018/19. 


Table C4. Average level of self-management by subgroup and grade 


p-value 
from test 
of equality 


across 


grades 
All 3.83 3.84 3.78 3.77 3.67 3.68 3.62 3.67 3.76 3.84 <0.001 
Gender a ae ay 
Male 3.73 3.74 3.69 3.73 3.65 3.65 3.55 3.62 3.70 3.80 <0.001 
Female 3.93 3.95 3.87 3.81 3.70 3.71 3.69 3.72 3.82 3.88 <0.001 
Absolute value of the 0.27*** 0.28*** 0.24*** 0.11** 0.07* 0.08* Ose" Ole Ole lal 
standardized 
difference in 
maximum and 
minimum value 


Survey year 
2017/18 3.80 3.81 3.76 3.75 3.71 3.66 3.58 3.63 3.74 3.86 <0.001 
2018/19 3.86 3.86 3.79 3.78 3.64 3.69 3.65 3.71 3.78 3.82 <0.001 


Absolute value of the 0.08** 0.07** 0.04 0.04 0.09** 0.04 0.09 0.11* 0.05 0.05 
standardized 

difference in 

maximum and 

minimum value 

Race/ethnicity 


Black 3.76 3.75 3.70 3.70 3.60 3.62 3.57 3.65 3.74 3.84 <0.001 
Hispanic 3.85 3.90 3.82 3.75 3.67 3.65 3.61 3.63 3.70 3.83 <0.001 
White 4.02 4.09 4.05 4.05 3.96 3.94 3.89 3.88 3.96 3.88 <0.001 
Other 3.91 3.97 3.97 4.00 3.90 3.85 3.91 3.81 3.90 3.79 0.084 


Absolute value of the 

standardized 

difference in 

maximum and 

minimum value 

At-risk status 
At risk 3.78 3.74 3.69 3.68 3.57 3.57 3.55 3.62 3.72 3.80 <0.001 
Not at risk 3.86 3.91 3.86 3.84 3.74 3.75 3.68 3.72 3.79 3.87 <0.001 

Absolute value of the 0.11*** 0.23*** 0.23*** 0.21*** 0.23*** 0.24*** 0.17** 0.13* 0.09 0.09* 

standardized 

difference in 

maximum and 

minimum value 


Ward 
1 3.79 3.73 3.82 3.64 3.68 3.57 3.51 3.64 3.68 3.84 <0.001 
2 3.79 3.84 3.68 3.86 3.73 3.80 3.93 3.93 3.87 3.84 <0.001 
3 3.93 4.02 3.94 3.90 3.78 3.80 3.77 3.72 3.82 3.81 <0.001 
4 3.89 3.92 3.82 3.76 3.72 3.68 3.57 3.62 3.68 3.81 <0.001 
5 3.72 3.78 3.68 3.72 3.57 3.66 3.65 3.62 3.76 3.83 0.002 
6 3.85 3.86 3.78 3.81 3.73 3.68 3.45 3.48 3.60 3.79 <0.001 
7 3.82 3.78 3.69 3.72 3.60 3.60 3.51 3.64 3.80 3.89 <0.001 
8 3.73 3.69 3.70 3.63 3.49 3.53 3.46 3.56 3.71 3.74 <0.001 
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Table C4. Average level of self-management by subgroup and grade (continued) 


p-value 
from test 


of equality 
across 
grades 


Absolute value of the 0.28*** 
standardized 
difference in 
maximum and 
minimum value 
Special education status 


Special education 3.63 3.61 3.60 3.61 3.51 3.47 3.41 3.47 3.50 3.79 <0.001 
Not special 3.86 3.88 3.82 3.80 3.71 3.72 3.67 3.72 3.81 3.85 <0.001 
education 


Absolute value of the 
standardized 
difference in 
maximum and 
minimum value 
English learner status 
English learner 3.81 3.82 3.70 3.65 3.66 3.61 3.52 3.63 3.63 3.81 <0.001 
Not English learner 3.83 3.84 3.79 3.78 3.67 3.68 3.63 3.67 3.77 3.84 <0.001 
Absolute value of the 0.03 0.03 0.12** 0.17** 0.01 0.09 0.15 0.05 0.19** 0.04 
standardized 
difference in 
maximum and 
minimum value 
Proficient/college ready in math and ELA 


0.29*** 0.25*** 0.27*** 


Proficient/ college 4.06 4.11 4.07 4.04 3.96 4.01 3.99 3.93 3.90 3.86 <0.001 
ready 555555 5S 
Not proficient/ 3.74 3.72 3.68 3.69 3.58 3.61 3.61 3.61 3.74 3.85 <0.001 


college ready 
Absolute value of the 
standardized 
difference in 
maximum and 
minimum value 
Chronic absentee status 


0.01 


Chronically absent 3.73 3.71 3.62 3.59 3.45 3.44 3.45 3.57 3.69 3.81 <0.001 
Not chronically 3.84 3.86 3.80 3.82 3.75 3.77 3.85 3.84 3.89 3.98 <0.001 
absent 


Absolute value of the 0.15*** 0.20*** 0.24*** 
standardized 
difference in 
maximum and 
minimum value 
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Table C4. Average level of self-management by subgroup and grade (continued) 


p-value 
from test 


of equality 
across 
grades 


Suspension status 
Ever suspended 3.38 3.36 3.36 3.44 3.36 3.38 3.33 3.43 3.44 3.67 0.001 
Never suspended 3.84 3.86 3.81 3.82 3.74 3.75 3.69 3.72 3.81 3.86 <0.001 


Absolute value of the 
standardized 
difference in 
maximum and 
minimum value 


Absolute value of the 


standardized difference Small Moderate Substantive eee | 


0.00—-0.09 0.10—-0.19 0.20-0.29 0.30+ 


ELA is English language arts. 

* Significant at p < .05; ** significant at p < .01; *** significant at p < .001. 

Note: The table shows the average level of self-management (described in appendix B) for each grade level, over both the full sample and for ten sets of 
subgroups. The means were calculated using nonresponse weights, as described in appendix B. 

Source: Authors’ analyses based on survey and administrative data provided by the District of Columbia Public Schools, 2017/18 to 2018/19. 


Table C5. Average level of sense of belonging by subgroup and grade 


p-value 
from test 
of equality 
across 
grades 
All 3.98 3.94 3.91 3.55 3.41 3.38 3.29 3.30 3.37 3.47 <0.001 
Gender 
Male 3.93 3.92 3.91 3.57 3.48 3.45 3.32 3.37 3.43 3.53 <0.001 
Female 4.02 3.96 3.92 3.52 3.33 3.30 3.26 3.22 3.30 3.40 <0.001 
Absolute value of 0.11*** 0.05* 0.01 0.06* 0.18*** 0.18*** 0.07 OMS Sear Or S eaten Oe ca 
the standardized 
difference in 
maximum and 
minimum value 


Survey year 
2017/18 3.95 3.92 3.90 3.55 3.42 3.37 3.29 3.29 3.38 3.47 <0.001 
2018/19 4.00 3.95 3.92 3.55 3.39 3.39 3.29 3.32 3.36 3.46 <0.001 


Absolute value of 0.06* 0.04 0.02 0.00 0.04 0.02 0.00 0.04 0.02 0.01 
the standardized 

difference in 

maximum and 

minimum value 

Race/ethnicity 


Black 3.95 3.88 3.89 3.54 3.38 3.38 3.27 3.29 3.36 3.49 <0.001 
Hispanic 4.07 4.07 3.99 3.60 3.50 3.40 3.32 3.32 3.38 3.39 <0.001 
White 3.94 3.95 3.91 3.47 3.36 3.36 3.38 3.37 3.43 3.44 <0.001 
Other 3.92 3.94 3.97 3.53 3.46 3.31 3.32 3.31 3.37 3.39 <0.001 


Absolute value of 0.18*** 0.22*** 0.12*** 0.15*** 0.16*** 0.11 0.13* 0.09 0.08 0.12* 

the standardized 

difference in 

maximum and 

minimum value 

At-risk status 
At risk 3.99 3.92 3.90 3.56 3.41 3.40 3.30 3.31 3.39 3.45 <0.001 
Not at risk 3.97 3.94 3.93 3.54 3.40 3.37 3.29 3.30 3.35 3.48 <0.001 

Absolute value of 0.02 0.02 0.04 0.02 0.01 0.04 0.01 0.01 0.05 0.04 

the standardized 

difference in 

maximum and 

minimum value 


Ward 
1 4.09 3.93 3.96 3.50 3.48 3.37 3.19 3.35 3.34 3.41 <0.001 
2 3.99 3.79 3.82 3.44 3.38 3.35 3.52 3.47 3.40 3.37 <0.001 
3 3.91 3.94 3.92 3.55 3.37 3.36 3.31 3.26 3.34 3.33 <0.001 
4 4.04 4.06 3.95 3.66 3.51 3.44 3.35 3.27 3.25 3.45 <0.001 
5 3.97 3.96 3.79 3.53 3.36 3.40 3.17 3.11 3.26 3.38 <0.001 
6 3.94 3.90 3.95 3.54 3.41 3.35 3.23 3.21 3.26 3.37 <0.001 
7 4.03 3.97 3.88 3.55 3.39 3.48 3.23 3.31 3.45 3.40 <0.001 
8 3.91 3.82 3.91 3.50 3.35 3.33 3.35 3.19 3.41 3.47 <0.001 
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Table C5. Average level of sense of belonging by subgroup and grade (continued) 


p-value 
from test 


of equality 
across 
grades 


Absolute value of 0.21*** On Gama 0.19** 
the standardized 
difference in 
maximum and 
minimum value 


Special education status 


Special education 3.93 3.84 3.85 3.60 3.49 3.36 3.27 3.38 3.34 3.58 <0.001 
Not special 3.99 3.95 3.93 3.53 3.39 3.38 3.30 3.28 3.38 3.44 <0.001 
education 


Absolute value of 0.07 0.13*** 0.09** 0.08 0.12* 0.02 0.04 0.12 0.05 0.16* 
the standardized 

difference in 

maximum and 

minimum value 

English learner status 


English learner 4.08 4.06 3.93 3.62 3.55 3.41 3.29 3.32 3.44 3.46 <0.001 
Not English 3.96 3.91 3.91 3.54 3.39 3.38 3.29 3.30 3.36 3.47 <0.001 
learner 


Absolute value of 0.14*** 0.18*** 0.02 0.09 0.19*** 0.04 0.00 0.02 0.09 0.01 
the standardized 

difference in 

maximum and 

minimum value 


Proficient/college ready in math and ELA 


Proficient/college 4.03 4.00 3.97 3.53 3.41 3.42 3.43 3.38 3.41 3.42 <0.001 
ready 
Not proficient/ 3.96 3.91 3.90 3.55 3.41 3.37 3.27 3.27 3.33 3.43 <0.001 


college ready 
Absolute value of 0.08*** 0.11*** 0.08*** 0.02 0.00 0.06 0.19*** 0.13* 0.09* 0.01 
the standardized 
difference in 
maximum and 
minimum value 


Chronic absentee status 


Chronically 3.93 3.88 3.81 3.49 3.30 3.27 3.24 3.28 3.35 3.47 <0.001 
absent 
Not chronically 3.98 3.94 3.93 3.56 3.44 3.42 3.37 3.34 3.42 3.46 <0.001 
absent 


Absolute value of 0.06 0.07* 0.14*** 0.08* 0.16*** 0.18*** 0.15*** 0.07 0.08* 0.01 
the standardized 

difference in 

maximum and 

minimum value 
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Table C5. Average level of sense of belonging by subgroup and grade (continued) 


p-value 
from test 


of equality 
across 
grades 


Suspension status 
Ever suspended 3.79 3.82 3.81 3.51 3.35 3.29 3.22 3.41 3.39 3.43 <0.001 
Never suspended 3.98 3.94 3.92 3.55 3.42 3.40 3.31 3.28 3.37 3.47 <0.001 


Absolute value of 0.22** 0.14* 0.13* 0.05 0.08 0.13** 0.11 0.15* 0.02 0.05 
the standardized 

difference in 

maximum and 

minimum value 


Absolute value of the 
standardized difference Small Moderate Substantive 


0.00-0.09 0.10—-0.19 0.20-0.29 0.30+ 


ELA is English language arts. 

* Significant at p < .05; ** significant at p < .01; *** significant at p < .001. 

Note: The table shows the average level of sense of belonging (described in appendix B) for each grade level, over both the full sample and for ten sets of 
subgroups. The means were calculated using nonresponse weights, as described in appendix B. 

Source: Authors’ analyses based on survey and administrative data provided by the District of Columbia Public Schools, 2017/18 to 2018/19. 


Table C6. Average level of social awareness by subgroup and grade 


p-value 
from test of 
Creel ia 


across 


grades 
All 4.01 3.96 3.88 3.75 3.60 3.61 3.50 3.58 3.67 3.76 <0.001 
Gender Sy 
Male 3.92 3.87 3.81 3.71 3.59 3.59 3.43 3.53 3.61 3.70 <0.001 
Female 4.09 4.06 3.96 3.80 3.62 3.63 3.58 3.63 3.74 3.83 <0.001 
Absolute value of 0.23*** 0.26*** 0.21*** 0.12*** 0.04 0.05 Onis 0.14* 0.18*** 0.18*** 
the standardized 
difference in 
maximum and 
minimum value 


Survey year 
2017/18 3.99 3.96 3.88 3.76 3.62 3.62 3.51 3.56 3.65 3.77 <0.001 
2018/19 4.02 3.97 3.88 3.74 3.59 3.60 3.50 3.60 3.69 3.75 <0.001 


Absolute value of 0.04* 0.01 0.00 0.03 0.04 0.03 0.01 0.05 0.05 0.03 
the standardized 

difference in 

maximum and 

minimum value 

Race/ethnicity 


Black 3.99 3.93 3.84 3.71 3.54 3.55 3.44 3.54 3.64 3.74 <0.001 
Hispanic 4.08 4.04 3.93 3.78 3.67 3.63 3.56 3.57 3.66 3.81 <0.001 
White 3.99 4.01 3.99 3.92 3.79 3.84 3.82 3.85 3.88 3.88 <0.001 
Other 3.97 3.95 3.94 3.78 3.70 3.67 3.68 3.75 3.82 3.81 <0.001 


Absolute value of GA" Oper 
the standardized 

difference in 
maximum and 
minimum value 


At-risk status 
At risk 4.01 3.95 3.83 3.70 3.54 3.54 3.42 3.53 3.66 3.72 <0.001 
Not at risk 4.00 3.98 3.92 3.79 3.65 3.65 3.58 3.63 3.68 3.80 <0.001 

Absolute value of 0.01 0.04 0.12*** 0.12*** 0.15*** 0.15*** [Opa 0.14* 0.03 0.11* 

the standardized 

difference in 

maximum and 

minimum value 


Ward 
1 4.06 3.89 3.94 3.56 3.58 3.50 3.43 3.56 3.58 3.81 <0.001 
2 4.02 3.95 3.85 3.77 3.60 3.64 3.92 3.92 3.86 3.90 <0.001 
3 3.96 3.99 3.96 3.85 3.69 3.74 3.69 3.71 3.75 3.76 <0.001 
4 4.10 4.06 3.88 3.82 3.70 3.62 3.45 3.57 3.66 3.84 <0.001 
5 4.03 4.00 3.82 3.77 3.59 3.61 3.51 3.45 3.62 3.68 <0.001 
6 3.97 3.96 3.90 3.75 3.63 3.62 3.36 3.40 3.52 3.69 <0.001 
7 4.02 4.01 3.79 3.67 3.53 3.51 3.25 3.44 3.60 3.71 <0.001 
8 3.93 3.84 3.88 3.67 3.43 3.46 3.31 3.40 3.63 3.55 <0.001 

Absolute value of 0.23*** (0, ey 


the standardized 
difference in 
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Table C6. Average level of social awareness by subgroup and grade (continued) 


p-value 
from test of 
equality 


across 
grades 

maximum and 

minimum value 

Special education status 


Special education 3.85 3.85 3.78 3.65 3.55 3.52 3.28 3.47 3.45 3.69 <0.001 
Not special 4.03 3.99 3.90 3.77 3.62 3.63 3.56 3.61 3.71 3.78 <0.001 
education 


Absolute value of 0.25*** 0.19*** 0.16*** 0.16*** 0.10 
the standardized 

difference in 

maximum and 

minimum value 

English learner status 


0.12 


English learner 4.06 4.01 3.86 3.71 3.69 3.56 3.48 3.58 3.62 3.83 <0.001 
Not English 3.99 3.96 3.89 3.75 3.60 3.61 3.51 3.58 3.67 3.76 <0.001 
learner 


Absolute value of 0.10** 0.07 0.04 0.05 0.12* 0.07 0.04 0.00 0.07 0.10 
the standardized 

difference in 

maximum and 

minimum value 

Proficient/college ready in math and ELA 


Proficient/college 4.06 4.05 4.00 3.86 3.74 3.81 3.82 3.74 3.84 3.89 <0.001 
ready 
Not proficient/ 3.98 3.93 3.84 3.72 3.56 3.57 3.50 3.50 3.64 3.74 <0.001 


college ready 
Absolute value of 
the standardized 
difference in 
maximum and 
minimum value 
Chronic absentee status 


0.11*** 0.16*** (ONpmee0.19*** (Olbms O27AE SO 21t es 


Chronically 3.94 3.90 3.79 3.65 3.47 3.45 3.35 3.50 3.61 3.73 <0.001 
absent 
Not chronically 4.02 3.98 3.90 3.78 3.65 3.67 3.72 3.71 3.79 3.91 <0.001 
absent 


Absolute value of 0.11* 0.11** 0.15*** 0.18*** Oma OAs OWE OAs 
the standardized 
difference in 
maximum and 


minimum value 
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Table C6. Average level of social awareness by subgroup and grade (continued) 


p-value 
from test of 


equality 
across 
grades 


Suspension status 

Ever suspended 3.67 3.65 3.56 3.57 3.39 3.39 3.20 3.43 3.49 3.63 <0.001 

Never suspended 4.02 3.98 3.90 3.78 3.65 3.66 3.58 3.61 3.70 3.78 <0.001 
Absolute value of OAS Oasr Oil 
the standardized 
difference in 
maximum and 
minimum value 


Absolute value of the 
standardized difference Small Moderate Substantive 


0.00—-0.09 0.10—0.19 0.20-0.29 0.30+ 


ELA is English language arts. 

* Significant at p < .05; ** significant at p < .01; *** significant at p < .001. 

Note: The table shows the average level of social awareness (described in appendix B) for each grade level, over both the full sample and for 10 sets of 
subgroups. The means were calculated using nonresponse weights, as described in appendix B. 

Source: Authors’ analyses based on survey and administrative data provided by the District of Columbia Public Schools, 2017/18 to 2018/19. 


Table C7. Average level of student satisfaction by subgroup and grade 


p-value 
from test 
of equality 


across 


grades 
All 4.04 4.02 3.93 3.72 3.53 3.49 3.50 3.44 3.48 3.50 <0.001 
Gender = eee 
Male 3.98 3.98 3.89 3.72 3.57 3.52 3.53 3.49 3.54 3.56 <0.001 
Female 4.10 4.05 3.97 3.71 3.49 3.44 3.48 3.39 3.41 3.45 <0.001 
Absolute value of 0.16*** 0.09*** 0.11*** 0.01 0.11*** 0.11** 0.07 Oley Oars (@aleye 
the standardized 
difference in 
maximum and 
minimum value 


Survey year 
2017/18 4.03 3.99 3.93 3.73 3.57 3.47 3.52 3.42 3.49 3.50 <0.001 
2018/19 4.04 4.04 3.93 3.71 3.49 3.50 3.49 3.47 3.47 3.50 <0.001 


Absolute value of 0.01 0.07* 0.00 0.03 0.11*** 0.04 0.04 0.07 0.03 0.00 
the standardized 

difference in 

maximum and 

minimum value 

Race/ethnicity 


Black 3.95 3.92 3.85 3.64 3.44 3.39 3.43 3.37 3.43 3.47 <0.001 
Hispanic 4.18 4.16 4.05 3.84 3.71 3.65 3.66 3.59 3.57 3.59 <0.001 
White 4.17 4.17 4.12 3.88 3.64 3.68 3.74 3.64 3.63 3.58 <0.001 
Other 4.06 4.09 4.04 3.79 3.58 3.60 3.62 3.54 3.55 3.52 <0.001 


Absolute value of 
the standardized 
difference in 
maximum and 
minimum value 
At-risk status 
At risk 3.98 3.96 3.86 3.67 3.48 3.40 3.45 3.44 3.48 3.51 <0.001 
Not at risk 4.08 4.06 3.99 3.76 3.57 3.54 3.55 3.45 3.48 3.50 <0.001 
Absolute value of = 0.13*** 0.13*** 0.17*** 0.12*** 0.12*** 0.18*** 0.13** 0.01 0.00 0.01 
the standardized 
difference in 
maximum and 
minimum value 


Ward 
1 4.10 4.02 4.03 3.61 3.62 3.50 3.46 3.44 3.43 3.46 <0.001 
2 4.11 3.94 3.90 3.77 3.55 3.54 4.01 3.86 3.79 3.65 <0.001 
3 4.16 4.15 4.09 3.90 3.63 3.67 3.55 3.44 3.48 3.45 <0.001 
4 4.11 4.15 3.94 3.82 3.65 3.58 3.62 3.58 3.46 3.60 <0.001 
5 4.05 3.99 3.85 3.64 3.47 3.40 3.38 3.24 3.36 3.38 <0.001 
6 3.99 3.97 3.95 3.60 3.38 3.26 3.27 3.22 3.26 3.34 <0.001 
7 3.96 4.00 3.77 3.54 3.32 3.52 3.41 3.39 3.36 3.25 <0.001 
8 3.90 3.82 3.87 3.64 3.46 3.32 3.28 3.16 3.43 3.31 <0.001 
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Table C7. Average level of student satisfaction by subgroup and grade (continued) 


p-value 
from test 


of equality 
across 
grades 


Absolute value of 
the standardized 
difference in 
maximum and 
minimum value 
Special education status 


Special education 3.96 3.94 3.87 3.68 3.56 3.45 3.38 3.42 3.41 3.53 <0.001 
Not special 4.05 4.03 3.94 3.73 3.52 3.49 3.54 3.45 3.49 3.50 <0.001 
education 


Absolute value of = 0.12*** 0.12*** 0.09** 0.07 0.05 0.05 0.21** 0.04 0.11 0.04 
the standardized 

difference in 

maximum and 

minimum value 

English learner status 


English learner 4.17 4.14 4.02 3.85 3.84 3.77 3.73 3.71 3.71 3.76 <0.001 
Not English 4.01 3.99 3.92 3.71 3.50 3.46 3.48 3.40 3.45 3.48 <0.001 
learner 


Absolute value of 0.21*** 0.20*** 0.13*** 0.18*** 
the standardized 

difference in 

maximum and 

minimum value 

Proficient/college ready in math and ELA 


Proficient/college 4.15 4.15 4.06 3.87 3.63 3.66 3.72 3.49 3.60 3.49 <0.001 
ready 
Not proficient/ 3.99 3.96 3.88 3.67 3.49 3.44 3.44 3.39 3.41 3.45 <0.001 


college ready 
Absolute value of 
the standardized 
difference in 
maximum and 
minimum value 
Chronic absentee status 

Chronically 3.92 3.91 3.78 3.58 3.38 3.34 3.40 3.41 3.46 3.50 <0.001 

absent 

Not chronically 4.06 4.03 3.95 3.76 3.58 3.54 3.66 3.50 3.52 3.51 <0.001 

absent 


OPA er een AaeeeO One 0.18*** Ooms 


Absolute value of 0.18*** 0.16*** 0.22*** 0.24*** 0.26*** 0.26*** 0.12** 0.08 0.01 
the standardized 

difference in 

maximum and 

minimum value 
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Table C7. Average level of student satisfaction by subgroup and grade (continued) 


p-value 
from test 


of equality 
across 
grades 


Suspension status 
Ever suspended 3.78 3.80 3.75 3.54 3.35 3.27 3.32 3.35 3.41 3.44 <0.001 
Never suspended 4.05 4.03 3.94 3.75 3.57 3.54 3.55 3.46 3.49 3.51 <0.001 


Absolute value of 
the standardized 
difference in 
maximum and 
minimum value 


Absolute value of the 


standardized difference Small Moderate Substantive ee 


0.00—-0.09 0.10—-0.19 0.20-0.29 0.30+ 


ELA is English language arts. 

* Significant at p < .05; ** significant at p < .01; *** significant at p < .001. 

Note: The table shows the average level of student satisfaction (described in appendix B) for each grade level, over both the full sample and for 10 sets of 
subgroups. The means were calculated using nonresponse weights, as described in appendix B. 

Source: Authors’ analyses based on survey and administrative data provided by the District of Columbia Public Schools, 2017/18 to 2018/19. 


For individual students who took the surveys in consecutive years, the year-to-year correlations between SEL 
competencies and school experiences were high, ranging from 0.36 to 0.47 (figure C2). However, these estimates 
were much lower than the year-to-year correlations of all of the academic proficiency and behavior variables, 
except for the number of times students were suspended. For example, the year-to-year correlations in students’ 
achievement test scores ranged from 0.79 to 0.86. Consistent with this finding, evidence has demonstrated that 
SEL competencies change more throughout the lifecycle than cognitive test scores, which stabilize earlier (Almlund 
et al., 2011). SEL competencies might be less stable because they reflect students’ behavior, which could also 
change with students’ circumstances (Heckman & Kautz, 2012). 


One question is whether the lower year-to-year correlations occur because they are measured with more error. 
Although this study does not have the data to investigate this possibility, other evidence suggests that 
measurement error is unlikely to account for most of the differences in year-to-year correlations between 
academic measures and SEL competencies and school experiences. A study of the CORE districts found that higher 
levels of measurement error of SEL competencies accounted for only a small part of the difference in their year- 
to-year correlations relative to achievement test scores (West et al., 2018). In addition, other research has shown 
that younger students are more susceptible to acquiescence bias—the tendency to report positive values on 
surveys (Soto et al., 2008). Acquiescence bias might explain why elementary school students report higher values 
of SEL competencies and school experiences than middle school students. However, the teacher reports of the 
measures also decline across grades, suggesting that acquiescence bias does not likely account for all of the 
differences across grades. Reference bias could also arise if students rate themselves using different benchmarks 
(Duckworth & Yeager, 2015). Further study could explore these measurement issues. 
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Figure C2. Students’ SEL competencies and school experiences were correlated between years but less so than 
most academic measures 


(a) SEL competencies & school experiences (b) Academic measures 
Student satisfaction 0.477" Fa 
ELA achievement 0.86""* 
Sense of belonging | MATES + 
pekemey oad Math achievement 
Social awareness | [CEI « 
Rigorous expectations | PT + In-seat attendance 
Settmanagement | NTRS + 
Number of suspensions 0.44*"* Ea 
Perseverance 0.36*** Fa 
00 01 02 03 04 05 06 07 08 09 1.90 00 01 02 03 04 05 06 07 08 09 10 
Correlation Correlation 


= Academic measures Si School experiences ae SEL competencies 


ELA is English language arts. SEL is social and emotional learning. 

*** Significant at p < .001. 

# Correlation differs from ELA achievement correlation by 0.10 or more—the cutoff for a moderate difference between correlations. 

Note: The figure shows the correlation between the 2017/18 and 2018/19 student reports of each SEL competency and school climate scale (described in 
appendix B), as well as between 2017/18 and 2018/19 data on four student academic measures. The correlations were calculated using nonresponse weights, 
as described in appendix B. 

Source: Authors’ analyses based on survey and administrative data provided by the District of Columbia Public Schools, 2017/18 to 2018/19. 


The year-to-year correlations of students’ SEL competencies and school experiences vary across subgroups of 
students, with differences ranging from low (0.00—0.09) to substantive (0.20—0.29) (table C8). For example, for 
each measure, the year-to-year correlations are lower by at least 0.02 for male students compared to female 
students. Similarly, the correlations are lower for students who receive special education services, are English 
learner students, are Hispanic or Black, and are not proficient/college ready in math and English language arts 
(ELA) than their subgroup counterparts. These differences in year-to-year correlations may reflect real differences 
in how some groups change, but they may also reflect measurement error. For example, students in some groups 
might find it challenging to understand items in the student survey or might select answers with less care. 
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Table C8. Year-to-year correlations of SEL competencies and school experiences 
Self- Self- Social Rigorous Sense of Student 
Perseverance management’ efficacy awareness expectations belonging satisfaction 


All 0.36 0.38 0.43 0.42 0.40 0.45 0.47 
Gender 
Male 0.31 0.34 0.39 0.37 0.36 0.44 0.42 
Female 0.42 0.42 0.48 0.46 0.45 0.46 0.52 
Absolute value of the 0.11 0.08 0.09 0.09 0.09 0.02 0.10 


difference in maximum 
and minimum value 
Race/ethnicity 


Black 0.34 0.35 0.42 0.39 0.39 0.44 0.46 
Hispanic 0.35 0.39 0.41 0.44 0.37 0.45 0.41 
White 0.47 0.50 0.54 0.48 0.50 0.52 0.53 
Other 0.46 0.42 0.48 0.55 0.49 0.51 0.51 
Absolute value of the 0.13 0.15 0.13 0.16 0.13 0.08 0.12 


difference in maximum 
and minimum value 
At-risk status 


At risk 0.35 0.33 0.40 0.38 0.38 0.42 0.42 
Not at risk 0.38 0.42 0.46 0.44 0.41 0.48 0.51 
Absolute value of the 0.03 0.09 0.06 0.06 0.03 0.06 0.09 


difference in maximum 
and minimum value 


Ward 
1 0.22 0.37 0.39 0.40 0.31 0.43 0.44 
2 0.48 0.47 0.52 0.50 0.47 0.54 0.49 
3 0.42 0.50 0.52 0.51 0.48 0.53 0.53 
4 0.38 0.41 0.43 0.44 0.39 0.44 0.47 
5 0.32 0.26 0.26 0.39 0.32 0.39 0.36 
6 0.40 0.39 0.46 0.43 0.45 0.48 0.51 
7 0.38 0.29 0.39 0.35 0.42 0.42 0.43 
8 0.30 0.34 0.42 0.36 0.38 0.41 0.41 
Absolute valueofthe | 026 O24 026 O16 O17 ®4«,.015 O17 
difference in maximum 
and minimum value 
Special education status 
Special education 0.26 0.24 0.33 0.30 0.24 0.35 0.37 
Not special education 0.40 0.42 0.46 0.45 0.45 0.48 0.50 
Absolute value of the 0.14 0.18 0.13 0.15 0.21 0.13 0.13 
difference in maximum 
and minimum value 
English learner status 
English learner 0.29 0.34 0.33 0.37 0.32 0.43 0.35 
Not English learner 0.38 0.39 0.44 0.42 0.41 0.45 0.48 
Absolute value of the 0.09 0.05 0.11 0.05 0.09 0.02 0.13 


difference in maximum 
and minimum value 
Proficient/college ready in math and ELA 


Proficient/college ready 0.47 0.51 0.52 0.49 0.47 0.51 0.52 
Not proficient/college 0.37 0.37 0.43 0.40 0.41 0.45 0.47 
ready 
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Table C8. Year-to-year correlations of SEL competencies and school experiences (continued) 


Self- Self- Social Rigorous Sense of Student 


Perseverance management’ efficacy awareness expectations belonging satisfaction 
Absolute value of the 0.10 0.14 0.09 0.09 0.06 0.06 0.05 
difference in maximum 
and minimum value 
Chronic absentee status 


Chronically absent 0.28 0.30 0.38 0.38 0.34 0.45 0.43 
Not chronically absent 0.41 0.41 0.46 0.42 0.42 0.45 0.48 
Absolute value of the 0.13 0.11 0.08 0.04 0.08 0.00 0.05 
difference in maximum 
and minimum value 
Suspension status 
Ever suspended 0.18 0.15 0.28 0.30 0.23 0.39 0.41 
Never suspended 0.40 0.41 0.46 0.42 0.42 0.46 0.47 
Absolute value of the 0.22 0.26 0.18 0.12 0.19 0.07 0.06 
difference in maximum 
and minimum value 


Absolute value of the difference 
between the maximum and 
minimum correlation Low Moderate Substantive 


0.00-0.09 0.10—0.19 0.20-0.29 0.30+ 


ELA is English language arts. SEL is social and emotional learning. 

Note: The table shows the pairwise correlation between the 2017/18 and 2018/19 versions of each SEL competency and school climate scale (described in 
appendix B) for the full sample and for nine sets of subgroups. The correlations were calculated using nonresponse weights, as described in appendix B. 
Source: Authors’ analyses based on survey and administrative data provided by the District of Columbia Public Schools, 2017/18 to 2018/19. 


Research question 2. To what extent do year-to-year changes in individual students’ SEL 
competencies and school experiences differ across schools? 


To help benchmark the difference in changes across schools, the study team calculated the average year-to-year 
changes in measures for average and high positive-change schools (those that are one standard deviation above 
average).”? The difference in changes between schools was comparable to or greater than the changes within 
either type of school. Consistent with the analyses showing that SEL competencies and school experiences decline 
across some grades, the year-to-year changes were slightly negative for average change schools in some cases 
(table C9). In contrast, for high positive-change schools, the changes were always positive. 


23 Approximately 16 percent of schools would be at least one standard deviation above average. 
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Table C9. Average year-to-year changes in measures for average and high positive-change schools 
Outcome units Percentile units 


High High 
Average positive- Average positive- 
Outcome change change change change 
units Xo aTeye)| Xo aYoye)| Difference Yo aToye)| Xo sTefe)| Difference 

Perseverance 1- to 5-point -0.03 0.08 0.11 -1.41 4.45 5.86 
scale 

Self-management 1- to 5-point 0.02 0.13 0.12 0.83 7.31 6.47 
scale 

Self-efficacy 1- to 5-point -0.02 0.15 0.17 -0.99 6.94 7.93 
scale 

Social awareness 1- to 5-point -0.02 0.11 0.13 -0.92 6.15 7.07 
scale 

Rigorous expectations 1- to 5-point 0.01 0.16 0.15 0.37 7.95 7.58 
scale 

Sense of belonging 1- to 5-point -0.03 0.15 0.18 -1.39 6.87 8.26 
scale 

Student satisfaction 1- to 5-point -0.07 0.08 0.15 -3.89 4.14 8.03 
scale 

Math achievement Standard 0.00 0.13 0.13 0.00 5.50 5.49 
deviations 

ELA achievement Standard -0.01 0.11 0.11 -0.29 4.36 4.65 
deviations 

In-seat attendance Fraction -0.03 0.08 0.11 -3.50 9.46 12.96 

Number of suspensions Number -0.12 0.08 0.19 -4.84 3.11 7.94 


ELA is English language arts. 

Note: This table shows the average year-to-year changes in measures for average change schools and high change schools (those that are one standard 
deviation above average). The values were calculated using nonresponse weights, as described in appendix B. 

Source: Authors’ analyses based on survey and administrative data provided by the District of Columbia Public Schools, 2017/18 to 2018/19. 


The intraclass correlation coefficients (ICCs) suggest similar conclusions about the extent to which year-to-year 
changes in the measures differ across schools as the metric presented in the main text (figure C3). Consistent with 
the findings presented in the main text, the ICC calculations suggest that changes in in-seat attendance vary the 
most across schools. However, changes in the SEL competencies and school experiences have similar but 
somewhat lower ICCs than changes in math and ELA achievement, which is the reverse of the percentile units 
presented in the main text. This difference arises because math and ELA achievement have higher year-to-year 
correlations, which reduces the differences in percentile units (see appendix B). 
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Figure C3. The ICCs for year-to-year changes in measures suggest similar conclusions as the percentile 


differences presented in the main text 
(a) SEL competencies & school experiences (b) Academic measures 
Perseverance 
Math achievement 
Self-management 


Self-efficacy ELA achievement 


Number of suspensions 0.04 


Social awareness 
Rigorous expectations 
Sense of belonging 


Student satisfaction 


o 


0.1 0.2 0.3 0.0 0.1 0.2 0.3 
Intraclass correlation coefficient Intraclass correlation coefficient 
Sj Academic measures (I School experiences a SEL competencies 


ELA is English language arts. ICC is intraclass correlation coefficient. SEL is social and emotional learning. 

Note: The figure shows the ICC of year-to-year changes in the measures across schools in 2017/18 (described in appendix B). The ICCs were calculated using 
nonresponse weights, as described in appendix B. The measures are defined in table B3. 

Source: Authors’ analyses based on survey and administrative data provided by the District of Columbia Public Schools, 2017/18. 


Compared to SEL competencies and school experiences, academic measures tend to vary more across schools, 
rather than within schools (figure C4). The ICCs for math achievement, ELA achievement, and in-seat attendance 
ranged from 0.32 to 0.49, whereas those for SEL competencies and school experiences ranged from 0.07 to 0.19. 
This finding is consistent with other research that has demonstrated that behavioral measures tend to have lower 
ICCs than achievement test scores (Schochet, 2008). 


Figure C4. The ICCs for most academic measures exceeded those of SEL competencies and school experiences 
(a) SEL competencies & school experiences (b) Academic measures 
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ELA is English language arts. ICC is intraclass correlation coefficient. SEL is social and emotional learning. 

Note: The figure shows the ICC of the measures across schools in 2017/18 (described in appendix B). The ICCs were calculated using nonresponse weights, 
as described in appendix B. The measures are defined in table B3. 

Source: Authors’ analyses based on survey and administrative data provided by the District of Columbia Public Schools, 2017/18. 
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Research question 3. How do measures of SEL competencies and school experiences relate to future 
outcomes, and how do they complement other available data for predicting future outcomes? To 
what extent do individual SEL competencies and school experiences relate to student outcomes 
measured one and two years later (such as achievement test scores, absences, suspensions, and 
whether a student feels loved, challenged, and prepared)? When other data are available—such as 
demographic information, achievement test scores, absences, and suspensions—to predict students’ 
future outcomes, to what extent does adding measures of SEL competencies and school experiences 
improve the predictive power and accuracy of those predictions? Which types of data and statistical 
models could best help DCPS classify whether students are at risk of having negative outcomes? 

The relationship between grade-3 SEL competencies and school experiences and grade-3 achievement in ELA is 
similar to findings for the full sample (table C10). As with the full sample, self-management is the most related to 
achievement in ELA and perseverance is slightly negatively related to achievement in ELA. 


Table C10. Pairwise correlations between grade-3 SEL competencies and school experiences and grade-3 ELA 
achievement 


Achievement Self- Social Rigorous Sense of Student 
measure Perseverance management _ Self-efficacy awareness Expectations belonging satisfaction 
Grade-3 PARCC -0.01 0.22 0.09 0.13 0.10 0.08 0.17 
ELA Score 
(2017/18) 
Grade-3 PARCC -0.06 0.17 0.01 0.03 0.03 0.01 0.08 
ELA Score 
(2018/19) 

Absolute value of the (iene 

correlation Low Moderate Substantive 

0.00-0.09 0.10-0.19 0.20-0.29 0.30+ 


ELA is English language arts. PARCC is Partnership for Assessment of Readiness for College and Careers. SEL is social and emotional learning. 

Note: This table shows the pairwise correlation between grade-3 PARCC ELA scores and individual SEL competencies and school experiences in 2017/18 and 
2018/19 (described in appendix B). The correlations were calculated using nonresponse weights, as described in appendix B. 

Source: Authors’ analyses based on survey and administrative data provided by the District of Columbia Public Schools, 2017/18 to 2018/19. 


SEL competencies and school experiences in grade 8 are related to measures of successful transitions to grade 9 
(table C11). All SEL competencies and school experiences are negatively correlated with credits behind in grade 9 
and positively correlated with successful grade progression. As in the full sample of students, self-management is 
the most related to successful outcomes. 


Table C11. Pairwise correlations between grade-8 SEL competencies and school experiences in 2017/18 and 
grade-9 outcomes in 2018/19 


Self- Self- Social Rigorous Sense of Student 

Grade-9 outcome Perseverance management’ efficacy awareness expectations belonging satisfaction 
Credits behind -0.05 -0.14 -0.13 -0.11 -0.12 -0.03 -0.07 
Progressed 0.02 0.07 0.00 0.05 0.02 -0.03 0.01 
successfully 

Absolute value of the 

correlation Low Moderate Substantive 

0.00-0.09 0.10-0.19 0.20-0.29 0.30+ 


SEL is social and emotional learning. 

Note: This table shows the pairwise correlation between grade-9 outcomes in 2018/19 and individual SEL competencies and school experiences in 2017/18 
(described in appendix B). The correlations were calculated using nonresponse weights, as described in appendix B. 

Source: Authors’ analyses based on survey and administrative data provided by the District of Columbia Public Schools, 2017/18 to 2018/19. 
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The study team estimated that credits behind in grade 9 accurately classifies eventual high school graduation 75 
percent of the time (table C12). The regression coefficients between credits behind in successive years were 
positive and close to 1 (0.98 and 1.02), demonstrating that students who are more behind in one year are also 
likely to be more behind in the next year. Credits behind was negatively associated with graduating from high 
school, indicating that students who were behind were less likely to graduate. 


Table C12. Results from analyses of the accuracy of classifying high school graduation on credits behind in 
grade 9 


Description of estimate Value of estimate 


Regression of grade-10 credits behind on grade-9 credits behind 


Intercept 0.30 

Coefficient 1.02 
Regression of grade-11 credits behind on grade-10 credits behind 

Intercept 0.28 

Coefficient 0.98 
Probit model of high school graduation as a function of grade-11 credits behind 

Intercept 1.20 

Coefficient -0.32 
Implied probit model of high school graduation as a function of grade-9 credits behind 

Intercept 0.97 

Coefficient -0.30 
Estimated classification accuracy of grade-9 credits behind in predicting high school graduation based on 0.75 
simulation 


Note: This table shows the output from a series of regression analyses designed to establish the relationship between credits behind in grade 9 and the 
probability of graduating high school (described in appendix B). 
Source: Authors’ analyses based on survey and administrative data provided by the District of Columbia Public Schools, 2017/18 to 2018/19. 


The correlations between predictors and outcomes one year out were similar to those between predictors and 
outcomes two years out (compare table C13 and table 1 in the main text). Compared to outcomes measured two 
years out, the correlations measured one year out are somewhat lower. However, the patterns of which variables 
are most predictive are similar, regardless of when the outcomes were measured. 
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Table C13. Correlations between predictors and students’ outcomes measured one year out 


Outcomes measured one year out in 2018/19 


Progressed successfully 


(grades 3-12) 
Graduated within 1 year 


Number of suspensions 
(grades 12) 


(grades 3-12) 

In-seat attendance 
Number of AP credits 
r-Taat-vom (:40-[e(-\- AO 7] 
Loved, challenged, and 
prepared (grades 3-12) 


Math achievement 
(grades 3-12) 


(grades 3-12) 
ELA achievement 


(grades 3-12) 
Credits behind 


(grades9-12) 


Predictor(s) in 2017/18 
(a) SEL competencies and climate perceptions 


Perseverance 0.05 0.04 -0.02 -0.03 0.09 0.00 0.08 0.04 0.28 
Self-management 0.05 0.13 0.24 0.25 0.11 0.17 0.10 0.13 0.24 
Self-efficacy 0.08 0.04 0.12 0.09 0.14 0.06 0.07 0.07 
Social awareness 0.07 0.11 0.12 0.13 0.16 0.13 0.07 0.08 
Rigorous expectations 0.10 0.06 0.07 0.07 0.20 0.03 0.05 0.04 
Sense of belonging 0.08 0.04 0.05 0.04 0.16 0.01 0.03 0.05 
Student satisfaction 0.07 0.09 0.13 0.12 0.17 0.05 0.03 -0.02 


(b) Academic measures 


Math achievement 
ELA achievement 0.17 0.17 
In-seat attendance 


0.28 
-0.18 


0.28 
-0.20 


Number of suspensions 
(c) Groups of variables 


SEL competencies and school 
climate perceptions 
Demographics 


Academic measures 
(d) Combinations of groups of predictors 


Demographics and academic 
measures 
All predictors 


Absolute value of the correlation Low Moderate Substantive High) 
0.00-—0.09 0.10-0.19 0.20-0.29 0.30+ 


AP is Advanced Placement. ELA is English language arts. SEL is social and emotional learning. 

Note: The table shows pairwise correlations and multivariate correlations between predictors in the left column and the outcomes in the top row. The 
outcomes were all recoded so that a higher value of the outcome is beneficial. The bold font indicates the correlation with the highest absolute value within 
the column and panel. The correlations were calculated using nonresponse weights, as described in appendix B. The sample included students who 
completed the Panorama survey in 2017/18. 

Source: Authors’ analyses based on survey and administrative data provided by the District of Columbia Public Schools, 2017/18 to 2018/19. 


The study team examined the correlations between school-level teacher and parent reports of perseverance and 
rigorous expectations and school-level outcomes one year later (table C14). Both perseverance and rigorous 
expectations were correlated with outcomes, but the relative strength of the correlations differed by 
respondent type. The teacher reports of perseverance were more positively correlated with outcomes than 
were the reports of rigorous expectations. However, the reverse was true of the parent reports for academic 
behaviors (suspensions and attendance) and whether students felt loved, challenged, and prepared. Overall, the 
teacher reports tended to be the most predictive. 
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Table C14. Correlations between school-level parent and teacher reports of perseverance and rigorous 
expectations and school-level outcomes measured one year out 


School-level outcomes measured one year out in 2018/19 


RYol YoYo) EI(=11/-)| Ko) =10 
predictor(s) in Progressed Number of Math In-seat challenged, and 
2017/18 successfully suspensions achievement ELAachievement attendance prepared 
Teacher reports 


Perseverance 
Rigorous 
expectations 
Parent reports 


Perseverance 


Rigorous 0.01 0.01 

expectations 

Low Moderate Substantive High | 
0.00-—0.09 0.10-0.19 0.20-0.29 0.30+ 


Absolute value of the correlation 


ELA is English language arts. 

Note: The table shows pairwise correlations between school-level predictors in the left column and school-level outcomes in the top row. The bold font 
indicates the correlation with the highest absolute value within the column. The school-level averages for teacher reports (but not parent reports) were 
calculated using nonresponse weights, as described in appendix B. 

Source: Authors’ analyses based on survey and administrative data provided by the District of Columbia Public Schools, 2017/18 to 2018/19. 


The classification accuracy for individual predictors and groups of predictors were consistent with the pairwise 
correlations and multivariate correlations presented in the main text (table C15). For all academic outcomes, 
individual academic measures accurately classified outcomes a higher percent of the time compared to individual 
SEL competencies or school experiences (compare panels a and b of table C15). Although the group of SEL 
competencies and school experiences improved the classification of academic outcomes relative to chance, they 
did so less than the group of demographic variables and academic measures (see panel c of table C15). Complex 
machine learning algorithms did not systematically perform better compared to probit models (compare panels c 
and d of table C15). Consistent with the multivariate correlations, predictive models that additionally included 
students’ SEL competencies and school experiences improved the overall accuracy little beyond the models with 
students’ demographic characteristics and academic measures (compare the last two rows in panels c and d of 
table C15). However, adding the SEL competencies and school experiences increased the accuracy by up to 7.8 
percentage points when classifying whether students felt loved, challenged, and prepared in the future. 
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Table C15. Classification accuracy of predictors 
Outcomes measured one year out in 2018/19 Outcomes measured two years out in 2019/20 


ge ready in math 
ge ready in ELA 
ly absent 
graduate 


z= 

3S 

a 
a 
a 
7) 
o 
° 
=] 
a 
wn 
rm) 
7) 
i 
iT) 
fe) 
- 
foe 


rex=vave(=ye| 
Not proficient/college ready in 
Not proficient/college ready in 
Red grade progression status 
Did not progress successfully 
grade progression status 


math 
Not loved, challenged, and 


prepared 
Not loved, challenged, and 


Chronically absent 
Did not earn AP credits 
Did not graduate 

a prepared 


Suspended 


Predictor(s) 
Accuracy from 
assigning students to 
the most prevalent 


<3 Did not earn AP credits 


Mae Did not 

© BT 

N Belarc) |=) 
S Bi eyaer)(2 
OG le tgeyatcer) |} 
Bye 

N Bye Mireys 


Ww 


4 70.1 93.6 8 


uw 


9 54.5 $93.3 92.2 72.9 


category 

a. Improvements in accuracy using SEL competencies and school experiences (probit) 

Perseverance 0.0 00 00 00 00 00 00 00 81 400 00 00 00 01 00 00 00 3.5 
Self-management 00 00 00 17 O14 00 00 00 71400 00 00 1.2 01 00 00 0.0 3.7 
Self-efficacy 00 00 00 00 03 00 00 00 99 $00 00 00 00 01 00 00 00 51 
Social awareness 0.0 00 00 00 0.7 00 00 00 97 $00 00 00 00 02 00 00 04 5.3 


Rigorous expectations 0.0 0.0 00 00 16 00 00 00 95 }00 00 00 00 0.77 00 00 02 3.1 
Sense of belonging 0.0 00 00 00 0.7 00 00 00 12.7)/00 00 00 00 02 00 00 0.0 6.7 
Student satisfaction 0.0 00 00 00 11 00 00 00 102])00 00 00 00 07 00 00 00 5.0 
b. Improvements in accuracy using academic measures (probit) 

Math achievement 00 0.0 15.55 22.9 2.0 4.7 00 04 01 $00 00 149 22.3 11 34 00 0.0 0.1 


ELA achievement 0.0 O.0 12.1 27.9 26 59 00 O08 0.4 00 0.0 13.4 256 1.2 45 0.00 0.0 0.2 
In-seat attendance 08 #00 O00 41 193 00 03 21 00 $1.3 01 0.77 43 13.33 00 07 41 40.3 
Number of 00 03 00 00 45 00 01 03 00 700 01 00 00 41 00 00 0.7 0.0 


suspensions 

c. Improvements in accuracy using groups of variables (probit) 

SELcompetenciesand 0.0 0.0 1.2 48 2.3 00 0.0 00 15000 00 08 49 13 00 00 00 89 
school experiences 

Basic demographics 0.0 O00 9.7 169 140 82 00 01 80409 00 86 164 90 94 00 02 5.7 
Academic measures 09 O3 164 288 192 59 O05 24 01 $15 02 16.3 263 132 43 01 46 0.0 
Basic demographics 1.2 O04 185 29.7 216 139 18 35 80 §2.0 02 17.6 27.7 15.7 148 03 52 59 
and academic 

measures 

All 1.3 0.4 18.6 29.7 21.7 14.0 2.0 3.7 15.8 92.0 O01 17.6 27.8 15.7 14.8 0.6 5.8 11.1 
d. Improvements in accuracy using groups of variables (random forests) 

SEL competenciesand 0.0 -0.1 1.6 5.7 19 -2.0 0.0 -0.3 12.9]-01 00 08 60 06 -1.3 0.0 -1.8 5.7 
school experiences 

Basic demographics 00 00 98 165 125 7.8 00 00 80 }07 00 85 15.5 87 9.7 00 0.0 54 
Academic measures 0.1 -0.4 15.7 27.7 16.7 5.6 -03 -06 -06 $03 -0.3 146 25.5 116 58 -0.2 04 -2.0 
Basic demographics 05 O38 18.1 29.4 20.4 14.0 -0.1 0.1 7.6 1.4 O.2 17.0 27.1 15.0 14.8 -0.1 19 6.1 
and academic 

measures 

All 0.2 O38 18.2 29.4 20.4 14.0 -0.2 16 14.2 91.2 O01 16.6 27.3 14.8 14.7 -0.1 13 9.0 


AP is Advanced Placement. ELA is English language arts. SEL is social and emotional learning. 

Note: Baseline accuracy is the accuracy achieved by predicting the most frequently occurring outcome for each student. The subsequent rows show the 
improvements in accuracy by using each predictor or group of predictor variables (see appendix B). 

Source: Authors’ analyses based on survey and administrative data provided by the District of Columbia Public Schools, 2017/18 to 2019/20 


When selecting a classification threshold, a receiver operating characteristic (ROC) curve can illustrate the tradeoff 
between accurately classifying students who eventually have positive outcomes (for example, graduating from 
high school) and inaccurately classifying those who eventually have negative outcomes (figure C5). For each 
model, each curve represents how the accuracy for these two groups of students changes as the threshold moves 
from 0 to 1. The dots on each curve represent the threshold that maximizes overall classification accuracy, which 
corresponds to the models used in this study. However, there are many potential decision rules to determine an 
optimal threshold for classifying observations. 


Any potential decision rule involves tradeoffs that should be evaluated in the context of the model’s intended 
usage. Reducing the threshold means that students who have a positive outcome are accurately classified a higher 
percentage of the time, but students with a negative outcome are accurately classified a lower percentage of the 
time. Depending on the relative importance of correctly or incorrectly classifying students with positive and 
negative outcomes, a different threshold may be appropriate. For example, for a model using demographic and 
academic predictors and a threshold that maximizes tota/ accuracy (0.55), approximately 88.8 percent of students 
who are not chronically absent are classified accurately, and 66.1 percent of students who are chronically absent 
are classified accurately (see figure C5). However, if DCPS placed a higher weight on accurately classifying students 
who are eventually chronically absent compared to those who are not (rather than total accuracy), then a different 
threshold might be more appropriate. For example, a threshold of 0.90 would lead to a classification accuracy of 
97.2 for those who are chronically absent but a classification accuracy of 34.2 for those who are not chronically 
absent. 


Consistent with figure 4, these findings suggest that the SEL competencies and school experience variables add 
little to the ability to accurately classify students who eventually have positive or negative outcomes across all 
possible thresholds (compare panels a and b of figure C5). The one exception is classifying whether students felt 
loved, challenged, and prepared in the future, a measure based on future SEL competencies and school 
experiences. For this outcome, the ability to accurately classify students with both negative and positive outcomes 
generally improves with the inclusion of these additional predictors. 
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Figure C5. A ROC curve illustrates how a model’s classification threshold governs the tradeoff between accurately classifying students with positive outcomes 


and misclassifying those with negative outcomes 


a. Demographic and academic predictors 


40.0 60.0 80.0 100.0 


Percentage of positive outcomes accurately classified 
20.0 


0.0 


20.0 40.0 60.0 80.0 
Percentage of negative outcomes inaccurately classified 


100.0 


Did not progress successfully 
Not college ready in ELA 


Red grade progression status 


Percentage of positive outcomes accurately classified 


20.0 40.0 60.0 80.0 100.0 


0.0 


Suspended 
Chronically absent 


Did not graduate 


b. All predictors 


20.0 40.0 60.0 80.0 
Percentage of negative outcomes inaccurately classified 


100.0 


Not college ready in math 


Did not earn AP credits 


Not loved, challenged, 
prepared 


AP is Advanced Placement. ELA is English language arts. 


Note: The figure shows the ROC curves for outcomes measured two years out using the probit model with the sets of predictors listed above the figures. The dots represent the threshold that maximizes overall classification 
accuracy. The diagonal, dashed line represents a random classifier. A point in the upper left-hand corner represents perfect prediction. 
Source: Authors’ analyses based on survey and administrative data provided by the District of Columbia Public Schools, 2017/18 to 2019/20. 


REL 2021-114 C-34 


Research question 4. How do measures of perseverance and rigorous expectations align across 
students, teachers, and parents? Across schools, to what extent do survey reports on these measures 
from students, parents, and teachers align? Is alignment associated with characteristics of schools 
(such as the demographic characteristics of their student population and the schools’ accountability 
ratings) and response rates on the survey? 

The alignment between student, parent, and teacher reports of perseverance and rigorous expectations differed 
across types of schools (tables C16—C19). In general, alignment was the lowest in schools with low School 
Transparency and Reporting Framework (STAR) ratings, a high percentage of Black students, and low student 
survey completion rates. The alignment also differed by geographic ward. 


Table C16. Difference between respondent types overall and by school characteristics for perseverance 


Teacher-student Teacher-parent Student-parent 
Difference in Standardized Difference in Standardized Difference in Standardized 
scale difference cYor=] (=) difference or-) [=] difference 


Ward 
1 -0.60*** 0.04 0.05 
2 -0.43*** | -0.40%** 0.03 0.04 
3 -0.19*** | -0.17** 0.03 
4 -0.63*** | -0.49%** 0.13* 
5 -0.84*** 0.22*** 
6 -0.59*** pares | 
8 -0.94*** 0.03 0.04 
Absolute value of 0.25 


the difference in 
maximum and 
minimum value 
STAR rating 


1 -0.98*** 0.07 0.09 
2 -0.87*** o.15*** [ag 
3 -0.70** oi" «O22 
4 -0.46*** 0.19*** 0.25 
5 -0.19*** 0.25 —— -0.20*** -0.26 0.01 -0.01 
Absolute value of 
the difference in 
maximum and 
minimum value 


Number of 16 - 224 16 - 227 16 - 225 
school-year 
observations 
Absolute value of the [ieeoreeterael| (feraen dl 
standardized difference Small Moderate Substantive 
0.00-0.09 0.10-0.19 0.20-0.29 0.30+ 


STAR is School Transparency and Reporting Framework. 

* Significant at p < .05; ** significant at p < .01; *** significant at p < .001. 

Note: The table shows the average school-level difference for reports on the 1- to 5-point perseverance scale (described in appendix B) between teachers 
and students, students and parents, and teachers and parents. 

Source: Authors’ analyses based on survey and administrative data provided by the District of Columbia Public Schools, 2017/18 to 2018/19. 


Table C17. Difference between respondent types overall and by school characteristics for rigorous 
expectations 


Teacher-student Teacher-parent Student-parent 
Difference in Standardized Difference in Standardized Difference in Standardized 
scale difference CY or-} (= difference scale difference 
037°" 
0.43*** -0.02 -0.03 
2 0.46*** | 0.aaeee | -0.03 -0.04 
3 0.41*** 0.11*** 0.14 
4 0.38*** 0.10* 0.13 
5 0.43*** 0.10* 0.13 
6 gaa" | 0.aeeee 0.14*** 0.18 
7 ages ous" 0.19 
8 0.27*** -0.01 -0.01 
Absolute value of 0.19 0.24 0.27 0.18 0.23 
the difference in 
maximum and 
minimum value 
STAR rating 
i oss" 0.00 0.00 
2 0.36*** | ager | 0.08* 0.10 
3 0.29*** 0.09* 0.12 
4 0.31*** o.10*** 0.13 
5 0.43*** | oageee | 0.07 0.09 
Absolute value of 0.26 0.16 0.20 0.10 0.13 
the difference in 
maximum and 
minimum value 
Number of 16 - 224 16 - 227 16 - 225 
school-year 
observations 
Absolute value of the aol 
standardized difference Small Moderate _ Substantive 
0.00-0.09 0.10-0.19 0.20-0.29 0.30+ 


STAR is School Transparency and Reporting Framework. 

* Significant at p < .05; ** significant at p < .01; *** significant at p < .001. 

Note: The table shows the average school-level difference for reports on the 1- to 5-point rigorous expectations scale (described in appendix B) between 
teachers and students, students and parents, and teachers and parents. 

Source: Authors’ analyses based on administrative data provided by the District of Columbia Public Schools, 2017/18 to 2018/19. 


Table C18. Correlation between school characteristics and average differences in responses for perseverance 


NET rele) (<1 Teacher-student Teacher-parent Student-parent 


Percentage of English learner -0.17** -0.12 -0.09 
students 
Percentage of special education 0.17* OMSiam 0.17* 
students 
Percentage of Black students 0.21** 
Percentage of Hispanic 0) Dee -0.14* -0.08 
students 
Percentage of White students -0.21** 
Student survey completion rate 0.01 
Number of school-year 223 - 224 223 - 224 224 - 225 
observations 

Absolute value of the ied 

correlation Low Moderate _ Substantive 

0.00-0.09 0.10-0.19 0.20-0.29 0.30+ 


* Significant at p < .05; ** significant at p < .01; *** significant at p < .001. 

Note: The table shows the pairwise correlation of average absolute school-level differences for perseverance (described in appendix B) between teacher 
and student reports, student and parent reports, and teacher and parent reports. 

Source: Authors’ analyses based on administrative data provided by the District of Columbia Public Schools, 2017/18 to 2018/19. 


Table C19. Correlation between school characteristics and average differences in responses for rigorous 


expectations 


Teacher-student Teacher-parent Student-parent 

Percentage of English learner -0.04 -0.13 -0.15* 
students 
Percentage of special education 0.05 -0.06 
students 
Percentage of Black students 0.01 0.04 OM ons 
Percentage of Hispanic 0.00 -0.08 -0.14* 
students 
Percentage of White students -0.02 0.03 =OP22 es 
Student survey completion rate SO R25 raat EO 2 Sanaa 
Number of school-year 223 - 224 223 - 224 224 - 225 
observations 

Absolute value of the Eeneral 

correlation Low Moderate Substantive 

0.00-0.09 0.10-0.19 0.20-0.29 0.30+ 


* Significant at p < .05; *** significant at p < .001. 

Note: The table shows the pairwise correlation of average absolute school-level differences for rigorous expectations scale (described in appendix B) between 
teacher and student reports, student and parent reports, and teacher and parent reports. 

Source: Authors’ analyses based on administrative data provided by the District of Columbia Public Schools, 2017/18 to 2018/19. 


Using nonresponse weights for students and teachers made little difference in the estimated alignment across 
student, parent, and teacher responses (figures C6 and C7). Relative to the analyses without weights, the 
correlations between respondent types changed by at most 0.02, and the average differences between 
respondent types changed by at most 0.01. This finding suggests that student nonresponse bias does not drive 
the results. However, the study team could not explore teacher and parent nonresponse, so it is not possible to 
rule out that the findings did not suffer from nonresponse bias due to those respondent types. 


Figure C6. Accounting for student and teacher nonresponse made little difference in the estimated 
correlations between respondent types 


(a) Perseverance (b) Rigorous expectations 

be Teacher & student 0.26" Ea £ Teacher & student SS. - 

a a. 
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** significant at p < .01; *** significant at p < .001. 

# Correlation differs from scale-specific “teacher & parent” correlation by 0.10 or more. 

Note: The figure shows the pairwise correlation of school-level averages of perseverance and rigorous expectations scales (described in appendix B) between 
teacher and student reports, student and parent reports, and teacher and parent reports. The school-level averages for students and teachers were 
estimated using nonresponse weights, as described in appendix B. 


Source: Authors’ analyses based on administrative data provided by the District of Columbia Public Schools, 2017/18 to 2018/19. 
SSS SS SS SS SSS SS SS SS SS SS SSS SSS eS J 
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Figure C7. Accounting for student and teacher nonresponse made little difference in the estimated average 
differences between respondent types 


(a) Perseverance (b) Rigorous expectations 
a # & Parent 4.15 
DS J 
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no] nol 
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# Absolute value of difference relative to students meets or exceeds 0.10. 

Note: The figure shows the average of school-level reports of perseverance and rigorous expectations scales (described in appendix B) for parents, teachers, 
and students. The school-level averages for students and teachers were estimated using nonresponse weights, as described in appendix B. 

Source: Authors’ analyses based on administrative data provided by the District of Columbia Public Schools, 2017/18 to 2018/19. 
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